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PREFACE 


This  paper  was  prepared  by  the  Institute  for  Defense  Analyses  (IDA)  for  the  Office 
of  the  Assistant  Secretary  of  Defense  (Production  and  Logistics)  under  a  task  entitled 
“Comparison  of  CAMS/REMIS  and  TICARRS.”  This  paper  serves  as  the  final  report  on 
IDA’S  evaluation  of  the  costs  and  operational  effectiveness  of  two  automated  Air  Force 
systems  for  maintenance  information.  One  of  the  systems  combines  the  Core  Automated 
Maintenance  System  (CAMS)  and  the  Reliability  and  Maintainability  Information  System 
(REMIS).  The  other  is  called  the  Tactical  Interim  CAMS  and  REMIS  Reporting  System 
(TICARRS). 

This  paper  was  reviewed  within  IDA  by  Bruce  N.  Angier,  Herbert  R.  Brown,  and 
James  L.  Wilson.  Before  this  paper  was  released  in  its  final  version,  the  Office  of  the 
Secretary  of  Defense  asked  interested  parties  both  inside  and  outside  the  government  to 
review  it  and  send  their  comments  to  IDA.  These  comments,  along  with  IDA’s  responses 
to  them,  are  contained  in  IDA  Document,  D-1400,  “Comments  on  IDA  Paper  P-2863,  ‘A 
Comparison  of  Air  Force  Data  Systems.’’’ 
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EXECUTIVE  SUMMARY 


This  paper  compares  the  costs  and  operational  effectiveness  of  two  automated 
systems  for  collecting  and  providing  maintenance  and  related  information  for  the  Air  Force. 
One  of  these  systems  (CAMS/REMIS)  combines  the  Core  Automated  Maintenance  System 
(CAMS)  and  the  Reliability  and  Maintainability  Information  System  (REMIS).  The  other  is 
the  Tactical  Interim  CAMS  and  REMIS  Reporting  System  (TICARRS). 

CONCLUSIONS 

In  Ida’s  judgment,  an  alternative  based  on  implementation  of  TICARRS  provides 
at  least  as  great  effectiveness  with  less  risk  and  less  cost  than  CAMS/REMIS.  In  terms  of 
effectiveness,  CAMS/REMIS  now  suffers  from  problems  with  availability, 
responsiveness,  and  data  integrity.  TICARRS  is  currently  missing  some  important 
elements  of  functionality  and  cannot  support  as  many  systems  and  kinds  of  equipment  as 
can  CAMS/REMIS.  In  our  cost  analysis,  we  included  the  cost  of  additional  work  needed  to 
overcome  the  shortcomings  of  both  systems.  We  remain  very  uncertain,  however,  that  the 
data  integrity  problems  of  CAMS/REMIS  can  be  fully  overcome.  Not  only  have  these 
problems  been  persistent,  but  the  CAMS/REMIS  architecture  works  against  their  simple 
resolution. 

Assuming  a  four-year  phase-in  period,  we  estimated  that  the  Air  Force  could  save 
$100  million  in  present  value  terms  over  the  next  ten  years  by  choosing  a  TICARRS-based 
system.  This  is  18  percent  of  what  we  estimated  it  would  cost  to  operate  a  CAMS/REMIS- 
based  system  over  that  period. 

METHOD  OF  EVALUATION 

We  evaluated  the  operational  effectiveness  of  CAMS/REMIS  and  TICARRS  as  they 
exist  today.  The  effectiveness  of  the  systems  was  compared  along  six  dimensions:  system 
functionality  (what  they  do),  scope  (for  what  systems  and  equipment),  operating 
characteristics  (their  availability,  responsiveness,  and  ease  of  use),  data  accuracy  and 
completeness,  adaptability,  and  logistics  and  operational  effectiveness  (measured  by  mean 
time  between  failures,  maintenance  man-hours  per  flying  hour,  mission-capable  rates). 
Limitations  of  the  systems  were  ictentified  through  a  review  of  documentation  and  extensive 
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discussions  with  users  at  all  levels  as  well  as  system  developers.  In  addition.  IDA  was  able 
to  use  the  Operational  Assessment  of  TICARRS  that  took  place  at  Seymour  Johnson  Air 
Force  Base  (AFB)  during  the  spring  of  1993  to  better  understand  its  strengths  and 
shortcomings.  To  the  extent  possible,  steps  required  to  overcome  the  limitations  of  each 
system  were  specified  and  cost  estimates  developed. 

The  ten-year  costs  of  the  systems,  modified  as  needed  to  improve  and  roughly 
equalize  their  effectiveness,  were  then  compared.  A  ten-year  time  period  was  used,  because 
it  seems  likely  that  newer  information  system  technology  should  displace  whatever  system 
is  chosen  in  about  ten  years. 

The  study  evaluates  two  alternatives  for  the  Air  Force  maintenance  information 

system: 

•  Alternative  1:  an  enhanced  version  of  CAMS/REMIS  that  incorporates 
improvements  that  are  either  under  way,  planned,  or  needed  in  the  near  future, 
and 

•  Alternative  2:  an  enhanced  version  of  TICARRS  that  is  expanded  in  function 
and  scope  to  allow  it  to  replace  CAMS/REMIS. 

OPERATIONAL  EFFECTIVENESS  OF  THE  CURRENT  SYSTEMS 

We  found  that  CAMS/REMiS  now  handles  a  greater  number  of  weapon  systems 
and  other  types  of  equipment  It  also  has  several  important  functions  that  ar'  missing  from 
TICARRS.  However,  we  also  found  that  TICARRS  operates  better  and  provides  more 
accurate  and  complete  data. 

Table  S-1  summarizes  IDA’s  findings  regarding  the  effectiveness  of  CAMS/REMIS 
and  TICARRS  as  they  exist  today. 


Table  S>1.  Comparison  of  the  Effectiveness  of  CAMS/REMiS  and  TICARRS 


Dimen.sion  of  Effeciivencs-s 
Functionality 
Scope 

Operating  characteristics 
Availability 
Responsiveness 
Ease  of  use 

Dau  accuracy  and  completeness 
Adaptability 

Logistics  and  opera 


Conclusion 

Greater  for  CAMS/REMIS 
Greater  for  CAMS/REMIS 

Better  for  TICARRS 
Better  for  TICARRS 

Equal  at  wing  level,  TICARRS  better  elsewhere 
Better  for  TICARRS 

TICARRS  better  in  short-term;  little  long-term 
difference 

No  difference  found 


itional  effectiveness 


Functionality 

CAMS/REMIS  was  found  lo  have  greater  functionality  than  TICARRS,  although 
there  are  some  functions  for  which  TICARRS  is  unique  (e.g.,  block  number  tracking). 
Among  the  more  important  functions  included  .m  CAMS/REMIS  but  not  in  TICARRS  are: 
interfaces  with  the  Standard  Base  Supply  System  and  the  Comprehen*svc  Engine 
Management  System,  and  handling  of  personnel  and  training  management  and  production 
scheduling  and  management. 

Scope 

CAMS/REMIS  was  found  to  support  far  more  aircraft  and  other  products 
(communications-electronics  equipment,  aerospace  ground  equipment,  etc.)  than  does 
TICARRS.  If  TICARRS  were  to  replace  CaMS/REMIS,  it  would  have  to  be  modified  to 
address  more  than  40  additional  aircraft  types  and  1,700  types  of  communications  and 
electronic  equipment. 

Operating  Characteristics 

Overall,  TICARRS  was  found  to  have  substantially  better  operating  characteristics 
than  CAMS/REMIS.  These  are  briefly  described  in  the  following  paragraphs. 

Regarding  system  performance  parameters,  TICARRS  and  REMIS  were  found  to 
provide  much  higher  levels  of  availability  than  CAMS.  TICARRS  and  REMIS  both  have 
had  less  than  one  percent  of  unscheduled  downtime.  CAMS  has  been  unable  to  approach 
its  target  of  95  peicent  availability  at  the  vast  majority  of  bases. 

TICARRS  was  found  to  have  a  better  response  time,  particularly  compared  to 
REMIS.  Some  evidence  indicated  that  TICARRS  was  able  to  produce  standard  reports  in 
less  than  30  minutes,  while  REMIS  averaged  2.4  hours.  Ad  hoc  REMIS  reports  took  an 
average  of  7.5  hours  and  in  some  cases  up  to  36  hours. 

Regarding  ease  of  use,  neither  system  was  found  to  be  convincingly  superior  at  the 
wing  level.  Except  for  some  users  at  depots,  other  users  above  the  wing  level  (Major 
Commands,  weapon  system  System  Program  Offices,  and  contractors)  emphasized  their 
problems  with  REMIS  and  showed  a  preference  for  TICARRS  when  they  were  familiar 
with  it. 
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Data  Accuracy  and  Completeness 

Where  comparisons  were  possible,  TICARRS  was  shown  to  have  more  complete 
and  accurate  data.  There  have  been  widespread,  persistent  failures  in  REMIS  receiving  the 
data  it  is  supposed  to  get  from  CAMS.  In  addition,  REMIS  has  been  rejecting  between  8 
and  1 1  percent  of  the  aircraft- related  information  (which  represents  one  of  the  most  critical 
types  of  equipment  tracked  in  REMIS)  it  receives  from  CAMS  because  of  data  errors.  For 
ail  equipment  types,  the  total  reject  rate  has  ranged  between  16  and  21  percent. 

Adaptability 

TICARRS  is  more  adaptable  in  some  ways.  The  Operational  Assessment  at 
Seymour  Johnson  showed  that  minor  modifications  to  the  system  could  be  made  quickly. 
TICARRS  appears  to  be  closer  than  CAMS  to  being  able  to  deploy  the  kind  of  stand-alone 
system  needed  for  wartime.  However,  neither  system  is  likely  to  have  a  distinct  advantage 
in  aiding  the  transition  to  the  next  generation  of  maintenance  information  systems. 

Logistics  and  Operational  Effectiveness 

We  found  no  statistically  significant  relationships  between  the  choice  of 
maintenance  information  system  and  aircraft  readiness  and  supportability  (as  measured  by 
mission-capable  rate,  mean  time  between  failures,  and  maintenance  man-hours  per  flying 
hour).  This  is  a  short-run  result  and  does  not  address  the  issue  of  using  the  information 
from  these  systems  in  longer-term  efforts  to  identify  bad  actors,  justify  modifications,  and 
so  on. 

EVALUATION  OF  COSTS 

The  total  ten-year  costs  (FY  1994-FY  2003)  of  the  systems  modified  as  needed  to 
improve  their  effectiveness  are  as  follows  (millions  of  FY  1994  dollars): 

•  CAMS/REMIS;  $645.63 

-  CAMS:  $502.93 

-  REMIS:  $142.70 

•  TICARRS:  $326.07 

The  two  sections  that  follow  explain:  (1)  the  factors  that  drive  those  costs  and  (2) 
the  cost  of  choosing  one  system  as  the  Air  Force- wide  maintenance  information  system. 
The  latter  includes  the  cost  of  operating  both  systems  until  the  chosen  system  has  achieved 
full  operational  capability. 
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Cost  Drivers 


The  cost  differences  that  result  from  our  comparison  of  CAMS/REMIS  and 
TICARRS  can  be  understood  by  identifying  and  evaluating  the  cost  drivers  for  the 
information  systems.  The  following  factors  greatly  affect  costs  in  maintenance  information 
systems: 

•  volume  of  application  software  (developed,  maintained)  and  size  and 
complexity  of  the  data  base, 

•  computer  operations, 

•  user  support  (both  on  and  off-site),  and 

•  communications. 

Table  S-2  shows  representative  annual  recurring  costs  for  CAMS,  REMIS,  and 
TICARRS  that  can  be  attributed  to  the  four  major  cost  factors. 


Table  S-2.  Comparison  of  Recurring  Annual  Costs  Across  Systems 


Major  Factors 

Cost  in  Millions  of  Dollars 

CAMS/REMIS 

CAMS 

REMIS 

Total 

TICARRS 

Softwaie/riata  Base 

$3.5 

$5.6 

$9.1 

$5.7 

Computer  Operations 

$10.1 

$2.4 

$12.5 

$4.0 

User  Support 

$18.4 

$1.1 

$19.5 

$8.1 

Communications 

$13.5 

$0.2 

$13.7 

$13.8 

The  table  shows  that  CAMS/REMIS  costs  more  than  TICARRS  in  three  of  the  four 
categories.  For  volume  of  software,  TICARRS  has  substantially  fewer  programmer- 
generated  and  maintained  lines  of  code  relative  to  CAMS/REMIS.  For  computer  operations 
and  user  support,  a  central  data  base  architecture  system  such  as  TICARRS  takes  much  less 
effort  to  operate  and  maintain  than  the  base-level  replicated  system  of  CAMS. 
Communications  costs  are  roughly  equal. 

Costs  of  Alternatives 

Table  S-3  shows  a  cost  comparison  between  the  two  alternatives  we  evaluated. 
Alternative  1  is  an  enhanced  version  of  CAMS/REMIS.  Alternative  2  is  an  expansion  of  the 
functionality  and  scope  of  TICARRS.  For  Alternative  2,  we  assumed  the  transition  to 
TICARRS  would  take  four  years.  The  “Alternative  2  With  Five-Year  Transition”  column 
represents  an  excursion  from  Alternative  2  with  a  one-year  delay  in  TICARRS  due  to 
lengthy  acquisition  procedures. 
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Table  8-3.  Total  Cost  Comparison  Between  Alternatives 


System 

Cost  in  Millions  of  FY  1994  I>ollars 

Alternative  1 
(CAMS/REMIS-hased) 

Alternative  2 
(TICARRS-based) 

Alternative  2  With 
Five-Year  Transition 

CAMS 

$502.93 

$154.98 

$203.45 

REMIS 

$142.70 

$48.00 

$59.71 

TICARRS 

$16.91 

$326.07 

$293.58 

Total 

$662.52 

$529.04 

$556.73 

Total  (Present  Value) 

$562.25 

$462.05 

$485.45 

Savings 

$133.48 

$105.79 

Savings  (Present  Value) 

$100.20 

$76.80 

Alternative  1:  An  Enhanced  Version  of  CAMS/REMIS 

Alternative  1  is  CAMS/REMIS  enhanced  to  improve  its  operating  characteristics. 
Under  this  alternative,  we  assumed  that  TICARRS  continues  for  18  months,  until  shortly 
after  REMIS  is  fully  operational.  We  estimated  that  this  alternative  would  cost  $663  million 
(in  FY  94  dollars)  over  the  next  ten  years  ($562  million  when  future  costs  are  discounted). 
The  major  cost  drivers  are  user  support  at  the  bases  and  computer  operations. 

Alternative  2:  An  Enhanced  Version  of  TICARRS 

For  this  alternative,  we  assumed  that  the  functionality  of  TICARRS  can  be 
expanded  to  match  that  of  CAMS/REMIS  by  the  end  of  FY  1995,  and  that  CAMS/REMIS 
would  continue  to  be  used  while  TICARRS  is  expanding.  Adding  the  required  functionality 
is  not,  in  our  judgment,  a  major  cost  driver.  The  major  cost  drivers  for  expanding 
TICARRS  are  in  initializing  the  data  base  for  each  new  weapon  system,  acquiring 
additional  hardware  to  handle  the  additional  work  load,  and  changing  the  individual  Air 
Force  units  from  CAMS  to  TICARRS. 

The  various  requirements  of  a  formal  MAISRC  acquisition  process  result  in 
TICARRS  being  fielded  in  place  of  CAMS/REMIS  no  earlier  than  the  fourth  quarter  of  FY 
1997.  An  alternative  based  on  TICARRS  was  estimated  to  have  a  ten-year  cost  of  $529 
million  ($462  million  when  discounted).  In  present  value  terms.  Alternative  2  would  be 
$100  million  cheaper,  a  savings  of  18  percent  relative  to  Alternative  1. 

If  the  policies  and  procedures  of  the  acquisition  process  require  more  time  to  change 
over  to  a  TICARRS-based  system  (i.e.,  five  years),  the  present  value  of  total  costs  to 
implement  TICARRS  would  be  $485.5  million,  compared  with  $562.3  million  for  the 
CAMS/REMIS  alternative  (Alternative  1).  This  represents  a  savings  of  $77  million  or 
14  percent 
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RISKS  FOR  EACH  ALTERNATIVE 


Given  the  resources  we  estimated  as  necessary  for  each  system,  we  assessed  the 
technical  risk  involved  in  the  enhancements.  We  judged  overall  risk  to  be  low  to  medium 
for  the  CAMS/REMIS  alternative  and  low  for  the  TICARRS  alternative. 

Both  CAMS  and  REMIS  have  efforts  under  way  to  address  their  shortcomings. 
Our  cost  estimates  included  additional  expenditures  to  further  improve  the  performance  of 
CAMS/REMIS.  Nevertheless,  some  aspects  of  the  design  and  history  of  CAMS/REMIS 
cast  doubt  on  the  ability  of  that  system  to  significantly  improve  performance.  The  move  to 
Regional  Processing  Centers  will  not,  by  itself,  improve  the  availability  of  CAMS. 
Improvements  to  availability  depend  on:  (1)  improved  software  quality  and  (2)  the 
provision  of  the  appropriate  mix  of  skilled  personnel  (data  base  managers)  at  the  bases. 
Moreover,  CAMS  will  continue  to  be  operated  in  an  environment  in  which  it  must  compete 
with  other  base-level  applications  for  computer  system  resources.  System  responsiveness 
is  likely  to  continue  to  suffer  in  the  future. 

It  should  be  possible  to  adequately  improve  the  responsiveness  of  REMIS. 
However,  we  believe  that  this  will  require  substantially  greater  resources  than  are  currently 
being  applied. 

The  biggest  potential  problem  involves  the  completeness  and  accuracy  of  CAMS 
and  REMIS  data.  The  fielding  of  the  Generic  Configuration  Status  Accounting  System 
(GCSAS)  may  improve  the  situation,  but  the  complex  nature  of  the  interface  between  the 
two  systems,  and  the  data  rejects  that  may  still  result,  could  yield  data  that  are  not  accurate 
enough  to  meet  some  of  the  Air  Force’s  requirements.  It  is  difficult  enough  to  enforce  data 
integrity  within  one  system.  CAMS/REMIS  may  be  programmatically  one  system  but  they 
are,  in  reality,  two  systems,  with  different  record  formats,  different  architectures  and 
different  edits. 

While  it  appears  to  be  possible  to  expand  TICARRS  in  functionality  and  scope  to 
perform  the  tasks  that  CAMS/REMIS  currently  supports,  a  major  expansion  is  not  without 
risks.  In  our  analysis  of  alternative  systems,  we  included  the  cost  of  expanding  TICARRS 
and  changing  over  all  Air  Force  units  from  CAMS  to  TICARRS.  Even  though  limitations 
in  scope  and  functionality  are  weaknesses  of  TICARRS  today,  the  flexibility  of  the 
system’s  architecture  supports  optimism  about  TICARRS’s  ability  to  be  expanded  as 
needed. 
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1.  INTRODUCTION 


A.  BACKGROUND 

This  paper  compares  the  costs  and  operational  effectiveness  of  two  automated 
systems  for  collecting  and  providing  maintenance  and  related  information  to  the  Air  Force. 
One  of  these  systems  (CAMS/REMIS)  combines  the  Core  Automated  Maintenance  System 
(CAMS)  and  the  Reliability  and  Maintainability  Information  System  (REMIS).  The  other  is 
the  Tactical  Interim  CAMS  and  REMIS  Reporting  System  (TICARRS). 

The  CAMS  segment  of  CAMS/REMIS  is  the  standard  Air  Force  base-level 
maintenance  information  system.  Its  development  began  in  the  early  1980s,  and  as  of 
December  1992  it  was  installed  at  107  Air  Force  bases.  It  supports  all  Air  Force  product 
groups  (aircraft,  engines,  missiles,  support  equipment,  test  equipment,  etc.)  and  functions 
within  maintenance  operations  at  the  base  level,  using  the  Standard  Base  Level  Computer 
(SBLC)  system.  It  also  supports  NATO’s  Airborne  Warning  and  Control  System  and  the 
Royal  Netherlands  Air  Force. 

The  REMIS  segment  of  CAMS/REMIS  is  the  Air  Force’s  central  data  base  for 
reliabili^  and  maintainability  (R&M)  data.  Its  objective  is  to  be  the  repository  for  all  Air 
Force  R&M  information  and  to  support  system  readiness  and  sustainability.  REMIS 
integrates  base-level  data  provided  by  CAMS. 

TICARRS  was  derived  from  the  Centralized  Data  System  (CDS)  developed  for  the 
F-16.  It  is  now  being  used  to  help  support  F-16s  and  many  F-15s.  A  version  of 
TICARRS,  called  Smart  Data  System  (SDS),  was  also  used  to  support  the  F-117A  until 
that  aircraft  switched  to  CAMS.  UCARRS  uses  a  central  data  base  architecture.  TICARRS 
used  to  receive  direct  input  of  base-level  data.  Currently,  base-level  data  are  supplied  to 
TICARRS  by  CAMS. 

The  fiscal  1992  Defense  Appropriations  Conference  Report  expressed  concern 
about  the  ability  of  the  CAMS/REMIS  systems  to  provide  timely,  accurate,  and 
comprehensive  data  to  Air  Force  system  managers.  The  General  Accounting  Office  (GAO) 
was  directed  to  make  an  independent  assessment  of  these  systems.  The  GAO  found  that 
several  base-level  studies  of  CAMS  indicated  that  problems  with  system  performance  and 
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data  accuracy  persisted.  The  GAO  also  found  that  REMIS  had  serious  software 
deficiencies,  many  of  them  leading  to  major  errors. 

The  Conference  Report  on  the  1993  Department  of  Defense  (DoD)  Appropriations 
Bill  supported  the  continued  operations  and  enhancements  of  CAMS/REMIS  and 
TICARRS  and  directed  the  DoD  to  have  the  Institute  for  Defense  Analyses  (IDA)  perform 
an  independent  analysis  of  the  systems.  This  paper  is  the  outcome  of  that  analysis. 

B.  PURPOSE 

The  purpose  of  the  IDA  analysis  was  to  compare  the  costs  and  benefits  of  the  two 
information  systems — CAMS/REMIS  and  TICARRS.  This  comparison  was  conducted 
within  the  framework  of  Air  Force  requirements  for  information  about  logistics  functions 
such  as  system  and  equipment  management,  maintenance,  and  configuration. 

In  part  to  respond  to  congressional  interest,  the  Air  Force  recently  undertook  an 
Operational  Assessment  of  TICARRS  92  at  the  4th  (F-15E)  Wing,  located  at  Seymour 
Johnson  Air  Force  Base.  TICARRS  92  is  the  most  up-to-date  version  of  TICARRS.  It  is 
similar  to  the  SDS.  The  Operational  Assessment  involved  directly  entering  base-level  data 
into  the  TICARRS  data  base  and  minimizing  the  use  of  CAMS  within  the  wing  for  six 
weeks.  Relevant  wing  personnel  were  trained  in  the  direct  use  of  TICARRS. 

Analysis  of  the  results  of  the  Operational  Assessment  of  TICARRS  92  was  part  of 
the  study  effort 

C.  APPROACH 

Our  approach  to  the  problem  was  to  thoroughly  evaluate  the  strengths,  weaknesses, 
and  differences  in  the  capabilities  of  CAM/REMIS  and  TICARRS.  Having  made  these 
evaluations,  we  defined  two  alternatives  with  essentially  equal  capabilities  that  meet  Air 
Force  requirements  for  a  maintenance  information  system.  One  alternative  is  based  on 
CAMS/REMIS  and  the  other  on  TICARRS.  The  study  provides  an  estimate  of  the 
recurring  and  nonrecurring  costs  of  each  alternative  over  a  ten-year  period. 

D.  OUTLINE 

This  paper  contains  eight  chapters  and  six  appendices.  Chapter  n  contains  an 
overview  of  CAMS,  REMIS,  and  TICARRS.  Descriptions  of  each  system’s  functions, 
hardware  and  software  configuration  and  management,  and  current  status  and  future  plans 
are  provided. 
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In  Chapter  III,  we  explain  some  considerations  that  underpin  the  approach  that  was 
used  to  evaluate  maintenance  information  systems.  In  addition  to  the  future  costs  implied 
by  a  particular  system,  we  considered  six  dimensions  of  effectiveness.  They  were: 

•  functionality  (What  does  the  system  do  and  not  do?), 

•  scope  (What  aircraft  and  kinds  of  equipment  does  it  handle?), 

•  operating  characteristics  (How  well  does  it  function?), 

•  data  accuracy  and  completeness  (How  good  is  the  information  it  produces?), 

•  adaptability  (How  well  could  it  accommodate  changing  environments  and 
technologies?),  and 

•  logistics  and  operational  effectiveness.  (How  does  it  affect  the  performance  of 
maintenance?  How  does  it  affect  the  ability  of  wings  to  meet  their 
commitments?) 

The  sources  of  information  used  to  analyze  the  alternative  systems  are  reviewed  in 
Chapter  IV. 

Chapter  V  contains  our  evaluation  of  CAMS/REMIS  and  TICARRS  as  they  exist 
today.  Five  kinds  of  activity  were  drawn  upon: 

•  extensive  review  of  literature  and  documents  dealing  with  CAMS,  REMIS,  and 
TICARRS; 

•  discussions  with  users; 

•  discussions  with  system  developers  and  maintainers; 

•  analysis  of  information  on  the  logistics  and  operational  effectiveness  of  aircraft 
wings  using  CAMS/REMIS  and  TICARRS; 

•  analysis  of  information  developed  as  part  of  the  Operational  Assessment  of 
TICARRS  92.  This  analysis  included  examination  of  responses  to  survey 
questionnaires  that  IDA  helped  develop  as  part  of  the  Operational  Assessment 
team.  It  also  incorporated  data  on  the  functional  capabilities  and  shortcomings 
of  TICARRS  92  developed  during  the  Operational  Assessment 

In  Chapter  VI,  we  describe  two  alternative  configurations  for  the  Air  Force’s 
maintenance  information  system.  One  is  based  on  CAMS/REMIS,  the  other  on  TICARRS. 
The  Air  Force  is  implementing  a  major  program  to  improve  the  operating  characteristics  of 
CAMS/REMIS,  and  we  took  the  likely  outcome  of  that  program  into  account  Similarly, 
we  accounted  for  the  fact  that  both  the  functionality  and  the  scope  of  TICARRS  would  have 
to  be  expanded  for  it  to  be  able  to  effectively  replace  CAMS/REMIS. 
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The  alternatives  considered  were: 

•  an  enhanced  version  of  CAMS/REMIS  that  incorporates  improvements  that  are 
either  under  way  or  planned  for  the  near  future  and 

•  an  enhanced  version  of  TICARRS  92  that  is  expanded  in  function  and  scope  to 
allow  it  to  perform  all  the  tasks  it  must  perform  to  replace  CAMS/REMIS. 

The  characteristics  of  the  systems  embodied  in  the  alternatives  (hardware,  software,  and 
support)  are  specified  in  some  detail,  and  the  likely  operating  characteristics  of  the  systems 
are  described. 

In  Chapter  Vn,  the  estimated  costs  associated  with  the  two  alternative  systems  are 
presented. 

Finally,  Chapter  vni  contains  our  conclusions. 

The  contents  of  the  six  appendices  are  as  follows:  Appendix  A  contains  the  results 
of  an  investigation  of  the  relationship  between  bad-actor  detection  and  data  accuracy; 
Appendix  B  lists  the  literature  reviewed  for  our  evaluation;  Appendix  C  contains  the  survey 
questionnaires  used  at  Seymour  Johnson;  Appendix  D  describes  our  test  of  REMIS 
functionality  and  selected  operating  characteristics;  Appendix  E  explains  our  vision  of  the 
future  Air  Force;  and  Appendix  F  describes  an  analysis  of  the  effects  of  the  systems  on 
weapon  system  performances. 
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11.  SYSTEM  DESCRIPTIONS 


This  chapter  contains  thumbnail  sketches  of  CAMS,  REMIS,  and  TICARRS  to 
give  readers  not  completely  familiar  with  them  an  introduction  to  the  subject.  Chapter  V 
contains  the  detailed  information  about  each  system  that  served  as  the  basis  for  the  IDA 
assessment  of  system  functionality  and  operation. 

CAMS  and  REMIS  combine  to  gather  and  provide  maintenance  information  for  the 
Air  Force.  Sometimes,  we  refer  to  them  as  a  single  information  system,  CAMS/REMIS. 
However,  considering  the  extent  to  which  they  differ  in  system  architecture,  hardware,  and 
management,  they  are  better  thought  of  as  two  systems  that  interact  with  each  other.  Our 
descriptions  treat  them  as  such. 

A.  CORE  AUTOMATED  MAINTENANCE  SYSTEM  (CAMS) 

1 .  Functional  Overview 

CAMS  is  the  standard  Air  Force  base^level  maintenance  information  system.  It 
supports  all  aircraft,  engines,  trainers,  support  equipment,  test  equipment,  missiles, 
munitions,  and  communications-electronic  maintenance  at  the  base  level.  CAMS  originally 
incorporated  and  enhanced  the  functions  of  its  precursors,  the  Maintenance  Management 
Information  Control  System  (MMICS)  and  the  Maintenance  Data  Collection  (MDC) 
systerrL  It  allows  on-line  information  entry  and  retrieval  from  a  data  base  that  is  integrated 
at  the  base  level. 

A  number  of  the  functions  supported  by  MMICS  and  MDC  were  functionally 
enriched,  converted  to  the  CAMS  environment,  and  released  with  the  initial  fielding  of 
CAMS  in  1985.  The  major  functions  provided  in  the  initial  fielding  were  as  follows; 

•  Job  Data  Documentation  (JDD).  This  subsystem  replaced  the  MDC  system  by 
providing  the  maintenance  community  with  a  complete  on-line  capability  to 
alter,  store,  and  retrieve  maintenance  information. 

•  Training/Personnel.  This  subsystem  is  used  to  produce  manning  rosters, 
listings,  and  course  codes.  It  links  people  to  requirements  and  provides  a 
training  forecast  capabUi^. 
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•  Inspection/Time  Change.  This  subsystem  tracks  equipment  and  time  change 
information,  allows  the  development  and  use  of  job-flow  packages  to  track 
items  through  the  inspection  cycle,  establishes  inspection/time  change  records 
fo'  each  piece  of  tracked  equipment,  and  provides  inspection/time  change 
forecasts. 

•  Status  and  Inventory  Reporting.  This  capability  allows  for  the  establishment  of 
an  initial  inventory  and  provides  automated  equipment  status  updating, 
equipment  status  information,  and  analysis  of  equipment  condition. 

•  Time  Compliance  Technical  Order  (TCTO),  This  is  used  for  entry  of  TCTO 
information,  reports  of  TCTO  status  changes,  and  inquiry  of  TCTO-related 
activities. 

•  Comprehensive  Engine  Management  System  (CEMS).  This  system  provides 
data  to  the  CEMS  data  base  in  the  appropriate  format 

Additional  functions  were  added  to  CAMS  over  the  period  1985-1992: 

•  Maintenance-Supply  Interface.  This  subsystem,  fielded  in  August  1990, 
automated  the  part-ordering  process  through  an  electronic  interface  with  the 
Standard  Base  Supply  System  (SBSS). 

•  Automated  Debriefing.  This  function,  fielded  in  October  1988,  provides 
automated  air  crew  debriefing  functions  and  automated  generation  of  the  Air 
Force  Technical  Order  (AFTO)  781  series  forms. 

•  Automated  Scheduling  and  Status  Inventory  Reporting  Systems,  This  function 
provides  the  mechanism  to  plan  and  schedule  equipment  use  and  maintenance 
on  a  monthly,  weekly,  or  daily  basis.  It  provides  equipment  status  and  location 
information  and  analyzes  equipment  condition.  The  automated  scheduling 
function  was  fielded  in  August  1988.  The  status  inventory  reporting  function 
was  fielded  in  August  1992, 

•  Personnel  Availability.  This  function,  fielded  in  August  1992,  automates 
information  on  cunent  personnel  availability  and  forecasts  future  availability. 

•  Automatic  Test  Equipment  Reporting  System  (ATERS).  This  subsystem 
provides  on-line  access  to  a  data  base  containing  organization,  equipment, 
status,  and  utilization  data  for  assigned  test  equipment  It  also  keeps  track  of 
the  capability  of  various  test  stations  to  test  particular  parts. 

•  Product  Quality  Deficiency  Reporting  Subsystem.  This  subsystem  reports 
known  or  suspected  deficiencies  for  equipment  weapon  systems,  or  related 
components  and  records  exhibit  disposition  instructions  and  data. 

•  REMIS  Interfaces.  This  series  of  functional  interfaces  is  designed  to  support 
REMIS. 
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2 .  System  Configuration  and  Management 

The  software  application  functions  for  CAMS  were  developed  and  are  currently 
maintained  by  personnel  at  the  Standard  Systems  Center  (SSC)  at  Gunter  Annex  to 
Maxwell  Air  Force  Base  (AFB),  Montgomery,  Alabama.  The  CAMS  Program  Office 
merged  with  the  REMIS  Program  Office  in  1991. 

The  CAMS  programs  collectively  consist  of  1.1  million  unique  source  lines  of  code 
and  are  written  in  COBOL.  The  programs  interface  with  the  Unisys  operating  system,  data 
base  manager,  and  communications  access  method.  CAMS  uses  a  network-hierarchical 
data  base  with  about  320  record  types  and  over  300  sets. 

The  data  base  varies  in  size  depending  on  the  activity  and  equipment  supported  at 
the  specific  base.  Typical  size  is  in  the  range  of  1.5  million  to  2.0  million  records. 

CAMS  operates  on  the  Standard  Base  Level  Computer  (SBLC),  which  is  a  Unisys 
2200/40x  series,  and  shares  that  system  with  other  base-level  data-processing  workload. 
CAMS  operates  at  109  active  bases  and  supports  153  National  Guard  and  Reserve  sites, 
the  Royal  Netherlands  Air  Force,  and  NATO  airborne  warning  and  control  aircraft  units. 

CAMS  supports  a  variety  of  terminal  types,  including  video  display  terminals, 
personal  computers,  terminal  printers,  and  remote  line  printers,  all  in  accordance  with  the 
SBLC  architecture. 

With  CAMS,  intra-base  communications  are  provided  by  the  communications 
configuration  of  the  particular  base.  Generally,  the  configuration  is  a  local  area  network, 
direct  connected  cables,  or  leased  lines.  Data  transfers  are  handled  under  proprietary 
protocols  (Unisys).  Inter-base  communications  occur  on  the  Defense  Data  Network  using 
Transmission  Control  Protocols/Intemet  Protocols  and  the  Automatic  Digital  Network. 

3.  Status  and  Plans 

a.  Status 

CAMS  achieved  full  operating  capability  in  August  1992.  The  following  functional 
capabilities  originally  included  in  the  functional  specifications  have  been  eliminated  or 
defied  until  requirements  are  revalidated: 

•  the  administrative/logistics  module  (not  required), 

•  follow-on  CEMS  interface  (deferred,  now  in  work), 

•  quality  assurance/quality  control  (tteferred),  and 

•  deployable  CAMS  (deferred). 


b.  Plans 


The  CAMS  program  management  at  Gunter  AFB  is  directing  its  programming 
resources  to  responding  to  the  problems  users  have  identified  with  the  system.  A  key 
development  activity,  referred  to  as  “CAMS  Optimization,”  incorporates  repairs  to  software 
defects  and  adds  help  screens,  simplified  data-entry  screens,  more  meaningful  error 
messages,  and  the  like.  The  plan  calls  for  improvements  to  be  available  for  installation  at 
the  base  level  in  four  quarterly  increments  between  April  1993  and  January  1994. 

In  addition,  CAMS  development  personnel  are  continuing  their  work  to  support 
REMIS,  including  interfaces  with  the  Generic  Configuration  Status  Accounting  System, 
which  will  not  be  fielded  until  after  its  Major  Automated  Information  System  Review 
Council  (MAISRC)  HI. 

The  CAMS  Program  Office  is  investigating  the  possibility  of  having  designated 
field  representatives  support  CAMS  users  in  the  future.  The  services  provided  by  these 
field  representatives  would  be  different  than  those  provided  by  the  current  base-level  data 
base  managers.  The  representatives  would  be  given  training  in  maintenance  operations  (so 
that  they  can  understand  the  use  of  the  CAMS  system  for  specific  maintenance-related 
functions)  in  addition  to  expertise  in  CAMS  systems  operations.  Their  role  would  be  to 
provide  on-site  support  to  CAMS  users  and  to  interact  with  CAMS  developers  to  improve 
system  functionality  and  performance. 

CAMS  development  will  be  completed  during  1993,  and  only  enough  technical 
personnel  will  be  retained  to  meet  operations  and  maintenance  needs.  It  is  expected  that 
additional  enhancement  will  need  to  be  funded  by  the  requesting  user  organization. 

B .  RELIABILITY  AND  MAINTAINABILITY  INFORMATION  SYSTEM 
(REMIS) 

1 .  Functional  Overview 

REMIS  is  an  integrated  data  base  system  that  is  meant  to  serve  as  a  centralized 
source  of  inventory,  status,  utilization,  configuration,  and  maintenance  data  for  all  Air 
Force  weapon  systems  and  support  equipment.  It  provides  a  central  data  base  architecture 
and  processing  environment  that  supports  Air  Force-wide  reliability  and  maintainability 
analyses,  as  compared  to  CAMS,  which  contains  maintenance  data  at  the  base  level  only. 

Current  maintenance  processes  and  procedures  require  the  use  of  twenty-three 
different  information  systems  at  Air  Force  Materiel  Command  (AFMC)  Headquarters,  the 
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Air  Logistics  Centers  (ALCs),  and  the  Major  Commands  (MAJCOMs).  REMIS  will 
replace  most  of  these  systems. 


REMIS  provides  the  capability  to  collect,  edit,  validate,  process,  store,  and  retrieve 
the  data  for  aircraft,  automated  rest  equipment,  communications-electronic  equipment, 
missiles,  and  other  items. 

REMIS  consists  of  three  subsystems  or  modules: 

•  Equipment  Inventory,  Multiple  Status  and  Utilization  Reporting  Subsystem 
(EIMSURS).  EIMSURS  contains  the  worldwide  inventory,  status,  and 
utilization  information  for  aircraft,  missiles,  trainers,  communications- 
electronics  equipment,  and  automated  test  equipment 

•  Product  Performance  Subsystem  (PPS).  PPS  is  the  central  source  for  all  Air 
Force  equipment  performance  data  derived  from  reportable  maintenance 
information. 

•  Generic  Configuration  Status  Accounting  Subsystem  (GCSAS).  GCSAS  will 
give  the  Air  Force  a  single  source  of  weapon  system  configuration  data.  Its 
major  functions  support  the  tracking  of  approved  and  actual  configurations  and 
Time  Compliance  Technical  Orders. 

2.  System  Configuration  and  Management 

REMIS  was  developed  by  Litton  Computer  Services  (LCS),  which  won  a 
competitive  contract  in  September  1986.  The  LCS  team  is  responsible  for  the  hardware, 
system  and  application  software,  data  base,  training,  communications,  maintenance,  and 
integration  support,  REMIS  consists  of  736,000  COBOL  source  lines  of  code  (data 
declarations  and  executable  lines,  excluding  comments).  REMIS  uses  Tandem  VLX 
computers  and  system  software,  including  the  Tandem  Guardian  90  operating  system  and 
the  Tandem  NonStop  SQL  data  base  manager.  REMIS  uses  a  centralized,  integrated, 
relational  data  base  with  over  100  million  records  on  line. 

REMIS  uses  five  regional  transaction  concentrator  centers  (Hill,  Kelly,  Robins, 
McClellan,  and  Tinker  AFBs),  four  of  which  have  two  Tandem  VLX  processors  and  the 
fifth  of  which  (Tinker)  has  five  Tandem  VLX  processors.  Each  of  the  centers  manages  user 
log-on,  authorization,  local  communication  functions,  and  screen  control.  REMIS  screen 
formats  are  stored  at  each  of  the  concentrators  to  reduce  transmission  time  from  the  central 
system. 
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The  REMIS  central  system,  located  at  AFMC  Headquarters  (Wright-Paiterson), 
uses  20  Tandem  VLX  processors  for  production.  Transactions  are  transmitted  to  and  from 
the  REMIS  central  system  using  52-ldlobit  communication  links. 

REMIS  terminals  are  personal  computers,  each  connected  to  one  of  the  regional 
concentrators  using  one  of  several  possible  types  of  connectivity  methods  (Ethernet,  Direct 
Connect,  dial-up,  local  area  network,  or  Telnet). 

3.  Status  and  Plans 

a.  Status 

The  first  component  of  REMIS,  EIMSURS,  was  fielded  in  September  1990.  The 
EIMSURS  and  PPS  components  have  reached  initial  operating  capability.  Full  operating 
capability  is  scheduled  for  June  1994. 

b.  Plans 

There  is  an  effort  under  way  to  improve  the  ad  hoc  report-generation  system  called 
REMISTALK.  In  addition,  the  possibility  of  improving  system  response  time  by  more 
evenly  distributing  the  processing  load  is  being  examined. 

C .  TACTICAL  INTERIM  CAMS  AND  REMIS  REPORTING  SYSTEM 
(TICARRS) 

1 .  Functional  Overview 

TICARRS  is  a  “fleet-wide”  centralized  data  base  that  combined  many  (but  not  all) 
of  the  types  of  functions  performed  by  CAMS  and  REMIS.  It  is  a  descendant  of  the 
Centralized  Data  System  (CDS),  which  was  first  developed  in  1979  under  the  sponsorship 
of  the  F- 16  System  Program  Office.  It  was  fielded  in  1982  as  a  direct-entry  and  -retrieval 
information  system  for  managing  the  maintenance  of  the  F-16.  Field  representatives  were 
added  in  1983.  During  1985,  the  Smart  Data  System  (SDS)  version  of  TICARRS  was 
developed  to  support  the  F-117A.  SDS  continued  to  support  the  F-117A  through  Desert 
Shield  and  Desert  Storm,  after  which  it  was  also  converted  to  direct  organizational-level 
entry  by  CAMS  as  other  systems  had  been  in  1989. 

TICARRS  now  receives  organizational-level  data  from  CAMS  and  provides 
weapon  system  performance  information  for  the  F-16,  F-15E,  and  Multi-Stage 
Improvement  Program  F-15  aircraft  to  various  users.  These  include  personnel  involved  in 
operations,  maintenance,  support,  and  program  management. 


In  its  direct-entry  form,  known  as  TICARRS  92,  TICARRS  combines  the  1987 
version  of  TICARRS  with  SDS.  TICARRS  92  includes  the  following  functions: 

•  Equipment  Inventory,  Status,  and  Utilization.  This  function  records  the 
ownership  and  possession  of  equipment,  as  well  as  its  mission  performance 
capability  and  recent  and  historical  utilization  rate. 

•  Flight  Scheduling.  This  subsystem  provides  for  scheduling  aircraft  sorties, 
modifying  and  deleting  them  as  appropriate. 

•  Support  of  the  Maintenance  Operations  Control  Center.  TICARRS  now 
includes  an  Automated  Maintenance  Operations  Center,  which  tracks  aircraft 
capability  and  other  conditions  critical  to  flight  line  and  aircraft  operations  for 
generating  sorties. 

•  Automated  Debriefing.  This  function  provides  for  direct  computer  entry  of 
information  on  aircraft  sorties  and  aircrew-reported  discrepancies. 

•  Maintenance  Job  Generation  and  Tracking.  This  function  generates  the 
maintenance  job  orders  to  be  performed  and  records  the  progress  on  these 
orders. 

•  Inspection/Time  Change.  Within  this  function,  TICARRS  schedules  phase 
inspections  as  well  as  Time  Change  Items  and  records  the  work  performed 
against  these. 

•  Time  Compliance  Technical  Orders  (TCTOs).  Within  this  function  are  entered 
the  work  to  be  done  under  each  TCTO  and  the  work  performed. 

•  Configuration  Management/Tracking.  This  function  provides  records  of  the 
items  that  are  actually  installed  on  a  specific  weapon  system  as  well  as  those 
items  formally  authorized  to  be  installed  on  that  equipment 

2.  System  Configuration  and  Management 

TICARRS  92  was  developed  and  is  maintained  and  operated  for  the  Air  Force  by 
Dynamics  Research  Corporation  (DRC).  It  incorporates  a  central  data  base  architecture. 

TICARRS  operates  from  two  Bull  DPS90  processors.  Communications  processors 
consist  of  a  Unisys  DCP30  used  over  leased  Sprint  lines.  A  GCOS8  operating  system  and 
a  TPS  transaction  processor  are  used.  The  data  base  uses  Integrated  Data  Storage  H,  is 
sized  at  3.4  gigabits,  and  has  about  22  million  on-line  records.  The  application  software  is 
in  COBOL  74  (approximately  417  thousand  unique  lines  of  code),  uses  a  productivity  tool 
called  Middleware,  and  has  batch-input-batch-output.  In  its  direct-entry  form,  TICARRS 
uses  the  base  communications  system  for  linkage  to  the  leased  lines  that  connect  to  the 
central  data  base. 


3.  Status  and  Future  Plans 


a.  Status 

TICARRS  enhancements  were  halted  in  1987  (with  the  version  called  TICARRS 
87),  but  enhancements  to  SDS  continued.  In  1989,  TICARRS  stopped  receiving 
organizational-level  information  directly  and  began  being  fed  data  from  CAMS  instead. 
However,  it  is  still  fed  directly  from  depots.  Product  Quality  Deficiency  Report  users, 
electronic  mail  users,  and  contractors.  The  continental  United  Stales  (CONUS),  is  divided 
into  six  regions  for  support  purposes.  In  addition.  Pacific  Air  Force  and  U.S.  Air  Forces 
Europe  are  separate  regions  that  use  satellite  communications  to  access  the  central  data 
base.  The  TICARRS  program  has  about  34  field  representatives  who  provide  support  to 
users.  Also,  the  CAMS/REMIS/TICARRS  programs  are  now  managed  by  a  single 
program  manager. 

An  Operational  Assessment  of  TICARRS  92  was  carried  out  at  Seymour  Johnson 
AFB  for  the  4th  Wing  (F-15E).  The  assessment  focused  on  how  well  TICARRS  can 
function  as  a  replacement  for  CAMS.  Wing  personnel  were  trained  in  the  use  of  TICARRS 
between  15  March  and  26  March  1993.  Starting  on  March  29,  users’  access  to  CAMS  was 
restricted  and  they  began  entering  data  directly  into  TICARRS.  This  continued  until  7  May, 
after  which  an  evaluation  was  performed.  IDA  provided  support  to  the  assessment  team, 
and  has  used  assessment  material  in  this  study. 

b.  Plans 

The  Air  Force  has  no  plans  to  enhance  TICARRS  in  any  way,  having  chosen 
CAMS/REMIS  as  its  standard  system.  DRC,  however,  has  analyzed  the  work  that  would 
be  needed  to  expand  TICARRS  to  give  it  functionality  and  scope  equivalent  to 
CAMSAIEMIS. 
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III.  EVALUATION  CONSIDERATIONS 


Alternatives  based  on  modifications  of  the  current  versions  of  CAMS/REMIS  and 
TICARRS  were  evaluated  on  the  basis  of  both  effectiveness  and  cost. 

A.  DIMENSIONS  OF  EFFECTIVENESS 

As  noted  in  Chapter  I,  IDA  addressed  seven  issues  in  examining  the  effectiveness 
of  the  alternative  systems.  They  were  functionality,  scope,  operating  characteristics,  data 
accuracy  and  completeness,  adaptability,  and  logistics  and  operational  effectiveness.  Each 
of  these  is  discussed  briefly. 

1 .  Functionality 

The  functionality  of  maintenance  information  systems,  such  as  TICARRS  and 
CAMS/REMIS,  is  the  list  of  actions  (or  operations)  that  they  perform  or  facilitate  to  help 
achieve  the  goals  of  maintenance  organizations  and  other  users.  Describing  those  actions  is 
not  an  easy  task.'  The  actions  are  derived  from  the  objectives  of  the  organizations 
supported. 

To  develop  a  framework  of  functions  an  information  system  should  provide,  IDA 
took  as  a  point  of  departure  a  1990  study  of  the  minimum  maintenance  management 
functions  required  in  a  deployed  environment  to  sustain  effective  aircraft  maintenance  sortie 
production.2  It  became  clear  after  a  review  of  the  TICARRS  and  CAMS/REMIS  functional 
descriptions  and  other  documents,  as  well  as  several  field  trips  and  interviews,  that  it 
would  be  necessary  to  break  out  in  more  detail  some  of  the  actions  included  in  that  study 
and  add  others  to  establish  an  appropriate  framework  for  analyzing  these  systems.  As  a 


'  A  reading  of  the  functional  descriptions  of  the  two  systems  clearly  reveals  that  the  difference  between 
them  is  a  matter  of  word  choice  in  some  instances  and  a  matter  of  substance  in  others. 

2  Air  Force  Logistics  Management  Center  Project  LM902028,  tepwted  in  Major  Scott  Taggart,  Mjgor 
John  Wood,  Captain  John  Renkas,  Captain  Jim  Martin,  C^>tain  Keith  Wynkoop,  Cq>tain  Russell 
Ellwood,  Chief  Master  Sergeant  Benjamin  Pate,  and  Mr.  Carroll  Widenhouse.  “Maintenance  Data 
Collection  Review  (On-Equipment  Failure  Data),”  Report  LM9 12082,  Air  Force  Logistics 
Managonent  Center,  Gunter  Annex  to  Maxwell  Air  Force  Base.  AL,  October  1991 . 
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result,  IDA  settled  on  a  list  of  17  major  functional  areas  to  compare  CAMS/REMIS  and 
nCARRS: 


•  equipment  inventory, 

•  equipment  status, 

•  equipment  utilization, 

•  flight  scheduling, 

•  support  of  the  Maintenance  Operations  Center, 

•  debriefing, 

•  maintenance  reporting/scheduling, 

•  maintenance  supply  system  interface, 

•  comprehensive  engine  management, 

•  cannibalizatioii  tracking/management, 

•  configuration  tracking/management  for 
-  aircraft  and 

~  engines, 

•  Time  Compliance  Technical  Order  (TCTO)  management, 

•  personnel  training/availability/scheduling, 

•  shop  production  planning/scheduling/control, 

•  mobilization  planning, 

•  system  deployability,  and 

•  other  features. 

While  this  list  is  fairly  extensive,  the  individual  categories  remain  general  and  can 
accommodate  a  wide  variety  of  specific  functions  within  their  boundaries.^ 


The  CAMS/REMIS/nCARRS  Program  Management  Office  recently  developed  a 
comprehensive  overview  of  required  functionality  that  incorporates  both  the  Air  Force’s 
stated  requirements  and  the  users’  stated  needs.^  That  overview  was  considered  in  the 


^  A  sucoessfu!  comparison  of  information  systems  requires  that  these  general  categories  be  expanded  into 
a  list  of  relevant  specific  functions. 

^  “System/Equipmoit  Reliability  and  Maintainability  (R&M)  Enterprise  Model,”  Cennity  Tedmologies, 
December  1992. 


development  of  the  list  of  functional  areas  and  provided  a  benchmark  for  our  analysis 
of  functionality. 

2.  Scope 

CAMS/REMIS  and  TICARRS  gather  and  manage  information  on  different  weapon 
systems  and  other  equipment.  Scope  refers  to  the  number  of  weapon  systems  or  types  of 
equipment  about  which  an  information  system  is  designed  to  accept  data.  Currently, 
CAMS/REMIS  has  greater  scope  than  TICARRS. 

An  important  objective  in  our  evaluation  of  TICARRS  was  to  understand  the 
difficulty  (and  expense)  that  would  be  involved  in  expanding  the  scope  of  TICARRS  to  be 
equivalent  to  that  of  CAMS.  The  Operational  Assessment  at  Seymour  Johnson  provided 
some  useful  experience  in  that  regard. 

3.  Operating  Characteristics 

The  effectiveness  of  maintenance  information  systems  extends  beyond  whether  they 
are  designed  to  handle  the  required  tasks  to  how  well  they  handle  them.  Indicators  of  the 
adequacy  of  a  system’s  operating  characteristics  include: 

•  system  availability — the  amount  of  scheduled  and  unscheduled  downtime; 

•  response  lime — the  speed  of  system  response  to  a  key  press  or  a  simple  query; 

•  turnaround  time — the  length  of  time  from  the  start  of  a  transaction  to  its 
conclusion;  and 

•  ease  of  use — this  has  many  aspects,  including: 

-  difficulty  in  learning  to  effectively  use  the  system, 

-  adequacy  of  training, 

-  presence  and  type  of  help  facilities, 

-  presence  of  error  checking,  completeness  of  checking,  and  usefulness  of 
error  messages, 

-  need  for  multiple  data  entry, 

-  ease  of  obtaining  reports,  both  standard  and  ad  hoc,  and 

-  consistency  of  the  user  interface. 

In  Chapter  V,  we  evaluate  information  system-based  measures  of  effectiveness  like 
these  in  our  analyses  of  the  competing  systems. 


4.  Data  Accuracy  and  Completeness 


The  principal  purpose  of  maintenance  information  systems  is  to  provide  users  with 
the  data  they  need  to  do  their  jobs;  however,  the  degree  of  data  accuracy  and  completeness 
that  is  needed  may  vary  across  users  and  the  supported  weapon  systems.  We  believe  that 
there  are  substantial  benefits  to  having  accurate  and  complete  maintenance  data.  These  data 
are  important  to  track  weapon  systems  in  their  operations  and  to  provide  needed 
information  on  system  performance  for  users  who  manage  and  repair  the  systems.  If  the 
data  are  not  accurate,  the  costs  to  operate  and  support  Air  Force  weapon  systems 
will  increase. 

No  system  will  yield  perfectly  accurate  and  complete  data.  When  automated  data  are 
suspect,  paper  backup  systems  are  sometimes  used;  however,  they  are  expensive  to 
maintain.  Among  the  benefits  of  automated  systems  are  lower  cost,  configuration  control, 
error  checking,  and  sharing  of  data  across  individuals  or  units.  Paper  systems  are  unlikely 
to  be  helpful  to  users  who  operate  at  a  different  level  than  the  data  are  kept. 

A  consequence  of  incomplete  or  inaccurate  data  is  that  Air  Force  personnel  make 
decisions  based  on  data  whose  inaccuracies  are  unrecognized  (uninformed  decisions),  or 
they  recognize  the  inaccuracies  and  choose  not  to  use  the  data,  making  decisions  with  little 
or  no  data  at  all.  They  may  be  unable  to  justify  a  course  of  action  needed  to  support  the 
weapon  system.  The  costs  and  adverse  consequences  of  such  problems  can  be  substantial. 

How  accurate  and  valid  does  the  information  need  to  be  to  support  management  and 
operational  decisions  in  the  Air  Force?  To  examine  this  issue,  we  address  the  stated  need 
for  accurate  and  valid  data  in  the  documentation  describing  existing  information  systems 
and  provide  an  independent,  but  subjective  assessment  as  to  how  accurate  the  data  need  to 
be  to  support  users  of  the  information  in  various  management  and  decision  processes  in  the 
Air  Force. 


a.  Requirements  Stated  in  Functional  Descriptions 

The  specific  performance  requirements  for  data  accuracy  and  validity  are  piesented 
in  the  functional  descriptions  for  CAMS  and  REMIS.  The  TICARRS  functional  description 
does  not  have  a  requirement  for  data  accuracy  and  validity.  In  CAMS  the  accuracy  and 
validity  requirements  are;  (1)  mathematical  calculations  will  be  accurate  to  the  fourth 
decimal  place  and  (2)  data  will  be  100  percent  accurate  within  the  limits  of  data  edits  for  the 
data  elements. 


For  REMIS  the  requirements  are  more  extensive: 

•  Mathematical  calculations — 100  percent  accurate  to  four  positions  right  of  the 
decimal  point,  when  the  number  is  expressed  in  scientific  notation. 

•  Data — 100  percent  accurate  for  data  elements  edited  by  table  look-up.  Other 
data  will  be  100  percent  accurate  within  the  capability  of  the  edits. 

•  Transmitted  data — ^no  less  accurate  than  allowed  by  the  Defense  Data  Network, 
which  has  a  probability  of  an  undetected  error  of  4.2  x  lO'l^. 

•  Data  validity — data  will  be  validated  before  entry  into  the  data  base.  At  a 
minimum  these  checks  will: 

-  check  data  previously  received  to  avoid  duplication, 

-  verify  data  elements  not  edited  at  the  input  source  to  ensure  appropriate 
alphabetic  or  numeric  data  are  contained  in  the  required  fields,  and 

verify  the  end-item  serial  number  against  the  master  inventory  file  to 
ensure  valid  serial  number,  equipment  designator,  and  possessing 
organization  is  being  reported. 

•  Statistical  confidence — confidence  inherent  in  the  variability  of  data  used  in 
appropriate  mathematical  calculations  and  standard  statistical  analyses  will  be 
addressed.  (Mean  time  between  failures,  mean  time  to  repair,  and  other  point 
estimates  will  have  associated  confidence  bounds  with  a  variety  of  statistical 
confidence  levels,  i.e.,  90  percent,  95  percent,  and  99  percent.) 

b.  Considerations  of  User  Requirements 

It  is  not  apparent  that  these  stated  performance  requirements  were  derived  from  a 
systematic  consideration  of  the  needs  of  various  Air  Force  and  industry  activities  for 
accurate  and  complete  data.  While  the  following  review  of  users’  requirements  is  not 
intended  to  be  the  definitive  treatment  of  this  subject,  it  does  provide  a  perspective  that 
needs  to  be  considered  in  evaluating  the  effectiveness  of  the  alternative  maintenance 
management  systems. 

We  conducted  interviews  with  senior  managers,  middle  managers,  and  system 
users  in  the  logistics,  acquisition,  operational,  and  industrial  communities  to  develop  an 
understanding  of  data  accuracy  and  validity  requirements.  We  assessed  requirements  to  the 
extent  possible  against  the  changing  Air  Force  environment:  the  movement  towards  two- 
level  maintenance,  which  places  a  premium  on  asset  visibility;  adoption  of  composite 
wings;  emphasis  on  small,  critical  force  structure  elements  (F-117  and  B-2);  and  the 
increasing  age  of  major  portions  of  the  force  structure. 
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For  some  applications  the  data  need  to  be  more  accurate  than  for  others.  To 
understand  the  consequences  of  data  accuracy  in  one  application,  we  investigated  the 
relationship  between  the  detection  of  bad-actor  line  replaceable  units  and  data  accuracy.  The 
analysis  develops  a  relationship  between  data  accuracy  and  the  time  required  to  detect  a  bad 
actor.  Time  can  be  measured  in  temporal  units  if  a  specific  maintenance  cycle  is  assumed  or 
in  number  of  maintenance  actions.  The  relationship  between  accuracy  and  time  to  detect  a 
bad  actor  is  negative.  Better  data  get  bad  actors  out  of  the  system  more  quickly.  Other 
studies  indicate  that  the  costs  impacted  include  base  labor,  depot  labor,  transportation,  and 
asset  investments.  Cost  savings  can  be  significant,  especially  for  high  cost  items  (avionics, 
engines).  The  details  of  the  analysis  are  presented  in  Appendix  A.  Without  a  more  complete 
investigation  of  the  costs  associated  with  different  levels  of  accuracy,  it  is  difficult  to  draw 
a  final  conclusion  from  the  analysis;  however,  the  analysis  does  indicate  that  a  range  of 
accuracy  between  70  to  95  percent  would  probably  be  sufficient  for  bad-actor 
identification.  For  mission  critical  items  (e.g.,  the  Inertial  Navigation  System  on  the 
F-117),  90  to  95  percent  accuracy  would  be  needed,  and  for  less  critical  items,  a  lower 
level  of  accuracy  would  probably  be  adequate. 

From  this  review,  IDA  has  developed  an  approach  to  data  accuracy  that  relates 
functional  activities  to  systems,  users,  specific  types  of  critical  information,  and  associated 
ranges  of  accuracy.  These  relationships,  shown  in  Table  III-l,  represent  a  subjective 
assessment  of  the  need  for  data  accuracy  from  the  users’  perspectives.  From  the 
information  presented  in  the  table,  it  is  clear  that  not  all  information  needs  to  be  available  to 
the  user  at  the  same  level  of  accuracy.  For  some  applications,  70  percent  data  accuracy 
might  be  adequate  to  meet  user  requirements.  However,  even  when  70  percent  is  sufficient, 
more  accurate  data  might  allow  the  user  to  define  a  problem  better  or  select  a  corrective 
action  more  rapidly,  resulting  in  reduced  cost  and  improved  operational  effectiveness. 
Based  on  this  subjective  assessment,  we  believe  that  a  maintenance  data  system  will  need  to 
operate  at  a  minimum  level  of  data  accuracy  of  about  90  percent 

For  selected  applications  (safety  of  flight,  mission  critical  equipment,  and  resource 
management),  data  accuracy  requirements  approach  1(X)  percent  When  a  small  number  of 
functions  require  more  accurate  data  than  others,  it  may  be  easier  to  give  additional 
management  attention  to  those  functions  than  to  increase  the  level  of  accuracy  in  the  entire 
data  system.  Equipment  status  and  readiness  data  can  be  made  highly  accurate  without 
imposing  the  same  requirement  on  failure  rate  information.  Mote  extensive  analysis  of  the 
costs  and  benefits  of  data  accuracy  could  be  very  valuable. 
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5 .  Adaptability 

During  the  lifetime  of  the  Air  Force’s  tactical  automated  maintenance  information 
system,  the  operating  environment  is  likely  to  change  in  many  significant  ways.  It  is 
important  to  evaluate  maintenance  data  systems  with  respect  to  their  ability  to  adapt  to  these 
changing  environments.  Areas  of  change  that  must  be  considered  are  described  in  the 
following  subsections, 

a.  Transition  to  War 

Since  it  is  the  job  of  the  Air  Force  to  be  ready  to  fight,  it  is  necessary  to  analyze 
TICARRS  and  CAMS/REMIS  in  terms  of  their  effectiveness  in: 

•  peacetime  conditions, 

•  mobilization  conditions  (e.g„  Desert  Shield),  and 

•  wartime  conditions. 

Mobilization  and  wartime  conditions  may  imply  reduced  access  to  information 
systems  based  in  the  continental  United  States  (CONUS)  and  increased  requirements  for 
deployable  communications  or  computer  hardware. 

b.  Technology 

Aircraft  technology  will  change.  In  particular,  the  built-in  ability  of  aircraft  to 
diagnose  the  nature  of  equipment  problems  will  probably  improve  substantially.  The 
information  system  should  be  able  to  accept  electronic  information  on  failure  modes  and 
circumstances  directly  from  the  aircraft  Similarly,  test  equipment  should  be  able  to  provide 
information  directly  to  the  information  system. 

Automated  maintenance  aids  that  use  expert-system  techniques  may  become  widely 
available,  replacing  technical  manuals.  In  some  cases,  these  aids  may  be  incorporated  into 
test  equipment  In  either  case,  they  would  benefit  from  direct  access  to  maintenance  history 
data  that  should  reside  in  the  maintenance  information  system. 

Computer  technology  will  also  improve.  The  automated  maintenance  information 
system  should  be  one  that  can  take  advantage  of  these  improvements  without  major  effort 
or  expense. 

In  addition,  the  need  to  interface  with  other  data  systems  may  change. 


c.  Maintenance  Procedures 

Air  Force  maintenance  procedures  may  change  in  ways  that  make  some  kinds  of 
information  more  important  than  they  used  to  be.  The  possible  move  toward  two-level 
maintenance  and  the  associated  requirement  for  asset  visibility  is  an  example  of  such  a 
change. 

The  Air  Force  is  currently  testing  the  feasibility  of  moving  towards  a  two-level 
maintenance  concept.  Two-level  maintenance  eliminates  the  intermediate-level,  or  back 
shop,  repair  process.  If  a  technician  cannot  make  the  required  repair  at  the  flight  line,  the 
component  is  sent  directly  to  the  depot  for  repair,  unless  a  quick  back  shop  screening 
shows  that  it  does  not  need  repair.  The  objective  of  the  two-level  maintenance  concept  is  to 
reduce  costs  (by  eliminating  much  of  the  back  shop).  If  transportation  times  are  such  that 
few  additional  assets  are  required  in  the  pipeline,  cost  savings  will  be  realized. 

Coronet  Deuce,  the  current  Air  Force  two-level  maintenance  test,  has  shown  that 
identification  of  bad  actors,  parts  that  continually  exhibit  the  same  failure  but  appear  to  be 
all  right  when  tested,  and  parts  that  are  believed  to  have  failed  but  really  have  not  are  critical 
to  the  success  of  two-level  maintenance.  Infonnation  on  the  maintenance  histories  of  parts 
is  the  only  way  to  make  the  identification.  The  ability  to  use  test  equipment  efficiently, 
facilitating  rapid  identification  of  bad  actors  and  usable  parts,  should  also  improve  the 
feasibility  of  two-level  maintenance. 

d.  Organization  of  Combat  Forces 

The  operational  organization  of  the  Air  Force  may  change  in  ways  that  make  some 
kinds  of  information  more  important  than  they  used  to  be.  The  adoption  of  composite 
wings  is  an  example  of  such  a  change.^ 

The  Air  Force  is  creating  a  number  of  composite  wings,  wings  with  single 
squadrons  of  different  kinds  of  aircraft.  The  idea  is  to  promote  integrated  operation  and 
facilitate  deployment  of  self-contained  combat  packages,  including  fighters,  bombers,  and 
tankers.  This  new  structure  is  a  departure  from  the  traditional  wing  structure,  which  is 
made  up  of  three  squadrons  of  the  same  kind  of  aircrafL 

The  composite  organization  may  pose  challenges  for  maintenance  and  for 
maintenance  information  systems,  particularly  in  the  context  of  two-level  maintenance. 


^  Both  two-level  maintenance  and  composite  wings  involve  perfonning  more  maintenance  at  locatimis 
otto  than  the  base  where  a  part  was  found  to  fail.  Easy  access  to  fleet-wide  maintenance  histery  data 
may  be  more  imptxtant  under  such  circumstances. 
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With  aircraft  of  a  given  type  more  widely  dispersed,  pans  will  probably  spend  less  time,  on 
average,  at  any  one  base.  This  could  make  base-specific  repair  histories  for  pans  less 
valuable  and  easy  access  to  fleet-wide  information  on  part  repair  histories  more  valuable. 

Our  study  considered  the  adaptability  of  both  information  systems  to  all  these 
dimensions  of  change. 

e.  Organization  of  Weapon  System  Management 

The  Air  Force  has  adopted  Integrated  Weapon  System  Management  (IWSM),  a 
concept  that  integrates  cradle-to-grave  management  responsibility  for  a  weapon  system  in 
one  place.  Under  this  concept,  maintenance  data  should  become  more  important  early  in  the 
development  process  because  greater  attention  will  be  paid  to; 

•  Estimating  operating  and  support  (O&S)  costs  early  in  the  process,  O&S  costs 
of  similar  systems  are  useful  in  this. 

•  Meeting  reliability  goals.  Developers  are  given  a  reliability  goal,  for  example,  a 
target  for  mean  time  between  failures  or  maintenance  man-hours  per  flying 
hour.  Maintenance  records  of  similar  existing  systems  help  the  government  to 
set  realistic  goals  for  the  new  system. 

Warranties  presumably  will  become  more  common  in  this  concept.  Maintenance 
data  systems  are  used  to  determine  when  contractors  receive  incentive  payments  or  must 
provide  free  parts  or  pay  penalties.  With  warranties,  accuracy  is  critical. 

Management  of  modifications  may  receive  greater  attention  under  IWSM,  Such 
management  requires  that  accurate  maintenance  data  be  fed  back  to  contractors  who  are 
producing  the  system  or  parts  of  the  system.  Simple  corrections  of  manufacturing  or  design 
defects  could  be  made  during  the  next  production  run,  if  flexible  manufacturing  fulfills  its 
promise. 

6.  Logistics  and  Operational  Effectiveness 

Improvements  in  reliability  and  maintainability  should  generate  improved  mission- 
capable  rates,  sortie-generation  rates,  ground-abort  rates,  and  air-abort  rates.  The  existence 
and  possible  extent  of  these  kinds  of  improvements  are  also  exambed  in  Chapter  V. 

A  key  presumption  underlying  the  Air  Force’s  stated  requirement  for  an  automated 
maintenance  data  collection  system  is  that  better  information  pays  dividends  in  terms  of 
better  logistics.  TTius,  identifying  problem  parts  and  getting  them  out  of  the  system  should 
improve  reliability.  Accurate  information  on  repair  histories  and  failure  modes  should 
reduce  repair  times.  Direct  interaction  between  the  maintenance  and  supply  information 
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systems  should  reduce  supply  response  limes.  Possible  relationships  between  the  choice  of 
maintenance  information  system  and  indicators  of  logistics  effectiveness  are  examined  in 
Chapter  V. 

B  .  CATEGORIES  OF  COST 

In  considering  the  costs  of  the  alternative  systems,  we  used  the  following  cost 
categories; 

•  Nonrecurring.  The  major  cost  elements  in  this  category  are  hardware 
acquisition  (including  hardware  to  support  system  development),  application 
software  development,  data  base  development,  acquisition  of  communications 
equipment  and  commercial  software,  system  integration  and  test,  and  user 
training. 

•  Recurring.  The  major  cost  elements  in  this  category  are  computer  operations, 
user  support,  and  maintenance  of  the  data  base  and  the  application  software. 

This  cost  structure  is  consistent  with  the  much  more  detailed  categorization  of  costs 
contained  in  the  Automated  Information  System  Cost  Element  Structure  that  is  incorporated 
in  the  Draft  Guidance  on  Automated  Information  Systems  Cost/Benefit  Analyses  from  the 
Office  of  the  Director,  Program  Analysis  and  Evaluation. 

Previous  expenditures  for  both  CAMS/REMIS  and  TICARRS  were  not  counted  in 
our  evaluation  of  competing  alternatives.  The  focus  was  on  those  portions  of  costs  over 
which  the  Air  Force  still  has  decision  power.  However,  information  regarding  past  costs 
was  used  to  assess  the  realism  of  estimates  of  future  costs. 
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IV.  SOURCES  OF  INFORMATION 


In  order  to  gain  a  thorough  understanding  of  the  characteristics,  costs,  strengths, 
and  weaknesses  of  the  information  systems  being  evaluated,  the  IDA  study  team  gathered 
information  from  various  sources.  Most  of  the  information  came  from  the  following 
activities:  review  of  the  literature  on  the  subject,  discussions  with  both  users  and 
developers  of  the  systems,  collection  of  logistics  and  aircraft  operations  data,  comparison 
of  CAMS  and  TICARRS  at  Seymour  Johnson  AFB,  and  assessment  of  REMIS 
functionality  and  selected  operating  characteristics.  These  activities  are  described  in  the 
following  sections. 

A.  REVIEW  OF  LITERATURE 

IDA  performed  an  extensive  review  of  the  literature  on  the  systems  being 
compared.  The  information  was  gathered  from  the  following  sources: 

•  the  Office  of  the  Secretary  of  Defense; 

•  the  Program  Management  Office  (PMO)  for  CAMS/REMIS  and  TICARRS ; 

•  other  Air  Force  activities,  including  Oklahoma  City  Air  Logistics  Center  and 
Standard  System  Center  at  Gunter  Air  Force  Base  (AFB); 

•  information  system  contractors,  including  Dynamics  Research  Corporation  and 
Litton  Computer  Systems; 

•  weapon  system  contractors,  such  as  Lockheed  Corporation;  and 

•  general  support  contractors,  such  as  Robbins-Gioia,  Incorporated,  Systems 
and  Applied  Sciences  Corporation,  Logistic  Systems  Architects,  and  Booz* 
Allen  &  Hamilton,  Incorporated. 

As  literature  was  received,  it  was  categorized  according  to  type  of  material,  information 
system  covered,  and  date.  A  list  of  the  documents  that  were  obtained  is  contained  in 
Appendix  B. 

The  types  of  material  were: 

•  information  system  documentation,  including  user's  manuals  and  functional 
descriptions; 

•  program  reviews  and  evaluations,  including  information  compiled  as  part  of 
the  Major  Automated  Information  System  Review  Council  (MAISRC)  and 
milestone  review  processes; 
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•  cost  estimates; 

•  briefings  and  presentations; 

•  correspondence. 

Information  was  further  categorized  according  to  the  system  or  systems  it 
addresses:  CAMS,  REMIS,  CAMS  and  REMIS,  TICARRS,  a  comparison  of  TICARRS 
with  CAMS  and/or  REMIS,  and  other  material  (such  as  discussions  of  future  requirements 
for  maintenance  data). 

B.  DISCUSSIONS  WITH  USERS 

Face-to-face  discussions  were  vital  to  understanding  how  users  interact  with 
information  systems.  The  IDA  study  team  traveled  widely  to  ensure  development  of  this 
kind  of  understanding. 

Users  visited  covered  the  Major  Command  (MAJCOM),  program  office, 
operational  (wing  and  squadron),  depot,  and  contractor  levels.  It  was  important  to  meet 
with  users  (and  advocates)  of  both  CAMS/REMIS  and  TICARRS.  The  activities  visited  are 
as  follows: 

•  Air  Combat  Command  Headquarters  (and  1st  Fighter  Wing — F-15s),  Langley 
AFB,  Virginia; 

•  F-15,  F-16,  and  F-22  program  offices,  Wright-Patterson  AFB,  Ohio; 

•  4th  Wing  (F-15Es),  Seymour  Johnson  AFB,  North  Carolina  (the  TICARRS 
92  Operational  Assessment  site); 

•  49th  Wing  (F-1 17 As),  Holloman  AFB,  New  Mexico; 

•  388th  Wing  (F-16s)  and  Ogden  Air  Logistics  Center  (ALC),  Hill  AFB,  Utah; 

•  Oklahoma  City  ALC,  Oklahoma; 

•  Sacramento  ALC,  California; 

•  Rockwell  Corporation,  Los  Angeles,  California; 

•  Lockheed  (formerly  General  Dynamics)  Corporation,  Fort  Worth  Division, 
Texas;  and 

•  McDonnell-Douglas  Coiporation,  St  Louis,  Missouri 

It  was  important  to  elicit  views  from  a  wide  range  of  user  communities  because 
different  kinds  of  activities  use  data  for  different  things.  Squadron  and  wing  maintenance 
personnel,  for  example,  largely  interact  with  maintenance  information  systems  in  order  to 
enter  data,  not  to  use  them,  although  some  repair  personnel  routinely  retrieve  data  on  the 
repair  history  of  parts.  Wing  engine  and  TCTO  management  functions  are  supported  by  the 
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data  systems,  as  is  the  tracking  of  aircraft  inventory,  utilization,  and  status.  The  MAJCOMs 
also  use  much  of  this  same  information. 

Depot  repair  personnel,  in  some  cases,  use  the  maintenance  information  to  assist 
with  repair  diagnoses.  Equipment  and  item  managers  use  the  systems  to  identify  persistent 
maintenance  problems  and  to  suggest  candidates  for  future  equipment  modifications. 
Program  offices  and  equipment  manufacturers  are  also  interested  in  pin-pointing  problems 
that  may  require  redesign  work. 

These  visits  gave  us  a  baseline  for  judging  how  the  information  systems  are 
currently  used,  how  effectively  they  function,  and  what  problems  they  have.  When  visiting 
users,  we  tried  to  gather  information  systematically  by  asking  the  following  questions: 

•  Which  information  system  do  you  use?  How  often  do  you  use  it? 

•  How  do  you  use  the  information  system  to  do  your  job?  How  much  do  you 
rely  upon  this  information,  verify  it,  supplement  it? 

•  What  standard  reports  do  you  generate? 

•  What  customized  reports  do  you  generate?  Do  you  have  examples  you  can 
share  with  us? 

•  What  functions  are  missing  from  the  system? 

•  What  operating  characteristics  are  good  or  bad  (timeliness,  accuracy, 
accessibility)? 

•  How  does  the  information  requirement  change  for  peace  versus  war?  How 
does  the  adequacy  of  the  system  change? 

•  What  resources  are  required  to  make  the  system  work? 

C.  DISCUSSIONS  WITH  DEVELOPERS 

Study  team  members  met  with  the  developers/maintainers  of  the  systems  in  order  to 
get  a  more  detailed  understanding  of  the  systems'  functions,  hardware  and  software 
architecture,  cost,  status,  and  problems.  All  the  relevant  information  system  developers 
were  visited  at  least  twice: 

•  Litton  Computer  Systems,  REMIS  developer,  Dayton,  Ohio; 

•  Standard  System  Center,  CAMS  developer,  Gunter  AFB,  Alabama;  and 

•  Dynamics  Research  Corporation,  TICARRS  developer,  Andover, 
Massachusetts. 

We  were  particularly  interested  in  the  cost  and  likely  effects  of  planned  system 
improvements.  We  used  as  a  general  guide  die  items  displayed  in  Table  IV- 1,  making 
appropriate  modifications  for  system-specific  variations. 
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System  architecture 


Haidware 


Ad^tability 


Reliability/Availability 


Securi 


Ease  of  Use 


Table  IV-1.  System-Related  Issues 


High-level  description 


Computer  _ 


Communications 


Tenninals 


g  system _ 


Key  utilities  (e.g.,  data  base  manager,  cwnmunications  package) 


Applications  programs  |  genesis,  age  (by  increment) _ 


language(s) 


size  (by  major  component/functicm) 


data  base  |  structure  (e.g.,  hierarchical,  rational) 


number  of  records 


record  size 


reporting/query  language 


,  cost,  schedule) 


Nature  of  previous  improvmnents  (sc 


Amount  of  scheduled  downtime 


Percent  up-time 


Failure  data 


Diagnostics  _ 


Protection/authorization  functions 


Trainin 


User  errors 


Screens  (number  and  organization) 


Data  elements 


Query/report  generation  tools 


Datadicdo 


Verification/modiftcation 


archival 


Hardware  and  sc^tware  maintenanoe 


rations 


Known  Rrobleins 


Planned  bnurovements 


Terminal 


Number  of  simultaneous  users 


Transaction  turnaround  tune 


DedkatecKshared 


Priorities,  schedulin 


measures/criteria 


Cost  and  schedule  firom  previous  increment 


Size  (lines  of  code) 


Eflbn  (staff-months) 


Duration  (months) 


don  each  major  improvemoit 


Estimated  cost  and  sdiedule 
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D.  LOGISTICS  AND  OPERATIONAL  DATA 


As  described  in  Chapter  in,  two  of  the  dimensions  of  effectiveness  used  to  evaluate 
CAMS/REMIS  and  TICARRS  were  the  effect  of  the  use  of  the  systems  on  logistics  and 
aircraft  operations.  Measures  of  logistics  effectiveness  include  failure  rates,  repair  times, 
and  required  asset  levels.  Measures  of  aircraft  operations  effectiveness  include  status 
indicators,  abort  rates,  and  utilization. 

IDA  received  data  on  such  measures  of  effectiveness  in  the  form  of  TICARRS  data 
on  F-15  and  F-16  aircraft  over  the  period  1982  to  1992.  In  addition,  the  CAMS/REMIS/ 
TICARRS  PMO  provided  IDA  with  a  detailed  time  line  of  events  pertaining  to  the 
implementation  of  and  changes  to  each  of  the  information  systems. 

IDA  used  these  data,  supplemented  by  information  from  the  Air  Combat  Command, 
to  analyze  logistics  and  operational  effectiveness  as  functions  of  the  information  systems 
being  used. 

E.  COMPARISON  OF  CAMS  AND  TICARRS  AT  SEYMOUR  JOHNSON 
AFB 

Seymour  Johnson  AFB  in  Goldsboro,  North  Carolina,  was  the  site  of  the  special 
six-week  Operational  Assessment  of  TICARRS  92,  beginning  March  29  and  ending  May 
7,  1993.  During  that  period,  Seymour  Johnson  personnel  moved  from  CAMS  to 
TICARRS.  This  assessment  provided  a  unique  opportunity  for  the  IDA  study  team  to  talk 
to  and  observe  the  same  users  first  with  CAMS  and  then  with  TICARRS.  The  following 
subsections  describe  IDA’s  activities  during  the  Operational  Assessment 

1 .  Interviews  and  Observations 

Before  the  TICARRS  92  assessment,  we  interviewed  CAMS  users  from  the 
following  work  centers  at  Seymour  Johnson: 

•  Debrief, 

•  Flight  Line  Maintenance, 

•  Plans  and  Scheduling  (wing  and  squadron  level), 

•  Avionics  Back  Shop, 

•  &igine  Management, 

•  Propulsion  Shop, 

•  Data  Base  Management, 
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•  Phase  Inspection 

•  Supply, 

•  Weapons, 

•  Annament, 

•  Analysis, 

•  Maintenance  Operations  Center,  and 

•  Training. 

Users  were  asked  to  describe  in  general  terms  their  responsibilities  and  their 
interaction  with  CAMS  (input  and  output).  They  were  also  asked  about  CAMS  functions, 
ease  of  use,  response  time,  and  availability. 

We  also  observed  users  from  the  same  work  centers,  except  Plans  and 
Scheduling  at  the  squadron  level  and  Training.  An  effort  was  made  to  observe  during 
periods  of  peak  interaction  (e.g.,  observing  Debrief  during  scheduled  landings).  Notes 
were  made  of  any  problems  such  as  exceptionally  long  response  times. 

We  returned  to  observe  these  same  user  groups  (and  when  possible  the  same 
users)  during  the  TICARRS  assessment  period. 

2.  Surveys 

The  TICARRS  92  Operational  Assessment  also  gave  us  the  opportunity  to  compare 
the  levels  of  satisfaction  of  a  given  set  of  users  with  CAMS  and  TICARRS  92.  To  assess 
user  satisfaction,  IDA  and  the  Air  Force  developed  the  1993  Air  Force  CAMS  User 
C^estionnaire  and  the  TICARRS  Assessment  (^estionnaire.  The  former  was  distributed 
before  the  start  of  the  assessment.  The  latter  was  administered  three  times — at  two,  four, 
and  six  weeks  into  the  assessment  Copies  of  the  questionnaires  are  included  as  Appendix 
C.  Wherever  possible,  the  questionnaires  were  constructed  using  the  same  language  in 
order  to  elicit  the  same  types  of  infoimation  about  both  CAMS  and  UCARRS  92. 

The  issues  addressed  in  the  questionnaires  included: 

•  background  information  on  the  respondent  including  pay  grade,  work  center, 
job  function,  education  level,  and  computer  experience; 

•  length  of  experience  with  the  system; 

•  frequency  of  system  use; 

•  duration  of  use; 

•  vorification  of  data; 


•  problems  with  the  system; 

•  criticality  of  the  system  to  the  respondent’s  job; 

•  satisfaction  with  ease  of  use  and  understandability; 

•  satisfaction  with  ability  to  input  and  output  data  quickly  and  easily; 

•  satisfaction  with  ability  to  modify  previously  input  data  easily  (a  consideration 
for  maintainers  who  make  an  initial  entry,  then  might  have  to  change  it  after 
making  a  final  diagnosis  or  deciding  exactly  which  parts  need  to  be  replaced); 

•  satisfaction  with  ability  to  use  the  system  to  perform  both  required  and  optional 
functions; 

•  satisfaction  with  accuracy  of  data  received; 

•  satisfaction  with  documentation; 

•  satisfaction  with  system  responsiveness;  and 

•  overall  satisfaction. 

The  questionnaires  made  extensive  use  of  response  options  that  constitute  a  Likert 
scale  (a  five-point  scale  often  used  for  survey  responses).  The  responses  to  questions  about 
level  of  satisfaction  included  “very  satisfied,”  “satisfied,”  “mixed/neither,”  “dissatisfied,” 
and  “very  dissatisfied.”  A  sixth  possibility,  “does  not  apply/don’t  know,”  was  also 
provided.  The  Likert  scale  is  desirable,  because  it  has  been  shown  that  these  options 
generate  proportional  responses;  that  is,  the  distance  between  a  “very  satisfied”  and  a 
“satisfied”  response  is  the  same  as  the  distance  between  a  “dissatisfied”  and  a  “very 
dissatisfied”  response. 

The  Air  Force  assessment  team  distributed  the  questionnaires  in  conjunction  with  its 
assessment  As  many  as  1,000  people  participated,  including  flight  line  maintainers,  data 
base  managers,  maintenance  operations  center  personnel,  and  other  key  users. 

IDA  also  helped  the  Air  Force  develop  a  survey  addressing  the  quality  of  the 
training  provided  at  the  beginning  of  the  assessment  The  training  survey  had  two 
functions:  (1)  to  provide  the  Air  Force  assessment  team  with  immediate  feedback  on  the 
adequacy  of  the  training  so  it  could  be  fine-tuned  as  appropriate,  and  (2)  to  provide  the  Air 
Force  and  IDA  with  information  on  the  effectiveness  of  training  (and  the  ease  of  learning  to 
use  TICARRS)  to  be  included  in  the  evaluation. 

To  achieve  the  first  purpose,  the  survey  asked,  for  example,  whether  or  not 
le^ndents  understood  that  they  were  expected  to  use  TICARRS  92  during  the  assessment 
and  whether  or  not  they  felt  able  to  use  it  To  achieve  the  second  purpose,  the  suney  asked 


about  satisfaction  with  the  amount  and  content  of  the  training  and  about  the  instructor’s 
knowledge  of  TICARRS  and  of  the  respondent’s  job. 

3.  Functional  Disconnects  and  Difficulty  Reports 

During  the  Operational  Assessment,  the  Air  Force  Assessment  Team  took  great  care 
to  document  user  problems  with  TICARRS  92.  Particular  attention  was  paid  to  the  ability 
of  TICARRS  to  provide  users  with  the  functionality  they  were  used  to  getting  from  CAMS. 
Users  experiencing  problems  filled  out  Functional  Disconnect  Work-Around  Write-Ups 
(FDWWs).  Over  400  FDWWs  were  filed,  although  many  of  them  did  not  reflect  problems 
with  TICARRS  functionality  but  were  caused  by  data  problems,  communication  problems, 
inadequate  training,  and  so  on,  and  the  same  problem  was  often  raised  multiple  times.  The 
FDWWs  were  an  important  source  of  information  for  our  comparison  of  CAMS  and 
TICARRS  functionality. 

The  FDWWs  were  supplemented  by  Difficulty  Reports  (DIREPs)  produced  by 
Dynamics  Research  Corporation  (DRC)  personnel  at  Seymour  Johnson.  The  DIREPs 
generally  identify  places  where  DRC  felt  it  would  be  useful  to  modify  TICARRS  to 
enhance  its  functionality  or  ease  of  use.  Most  of  the  DIREPs  fell  into  the  category  of  bugs 
or  minor  modifications  to  system  parameters.  Many  of  the  DIREPs  led  to  modifications 
being  made  to  the  system  during  the  Operational  Assessment.  Several  areas  of  major 
missing  functions  (e.g.,  production  scheduling)  were  identified,  but  not  fixed  during  the 
Operational  Assessment.  The  costs  of  fixing  them  were  addressed  in  our  study. 

F.  TEST  OF  REMIS  FUNCTIONALITY  AND  SELECTED  OPERATING 
CHARACTERISTICS 

IDA  conducted  a  detailed  assessment  of  the  current  functionality  and  status  of 
REMIS  to  compare  the  data  output  capabilities  of  TICARRS  and  REMIS.  We  gave  the 
CAMS/REMIS  Program  Management  Office  and  the  F-15  System  Program  Office 
(TICARRS  users)  two  identical  data  requests  in  late  May  1993.  The  first  data  request 
(shown  in  Figure  IV- 1)  gave  three  working  days’  lead  time;  the  second  (shown  in  Figure 
IV-2)  was  given  upon  our  arrival  at  the  respective  Dayton,  Ohio  offices.  The  requests 
covered  several  aircraft  types,  and  the  data  requested  are  typical  of  those  we  had  seen 
requested  by  base,  ALC,  and  MAJCOM  users.  The  test  focused  on  both:  (1)  the  basic 
functions  of  REMIS  and  (2)  specific  areas  of  functionality  that  TICARRS  users  have 
requested  be  provided  before  TICARRS  would  be  fully  replaced  by  CAMS/REMIS.  A 
m<»e  detailed  discussion  of  the  assessment  and  its  results  is  contained  in  Appendix  D. 
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REMIS  Data  Request — 21  May  1993 

1.  Monthly,  October  1992  through  AjHil  1993,  forF-15E  aircraft,  4th  Wing: 

A.  Mission-capable  rates  number  and  percent  of  aircraft  FMC,  PMC,  and  NMC 

B.  MTBF  and  MTBMA  by  two-digit  WUC 

C.  Re-test  OK  rates  by  two-digit  WUC 

D.  MMH/FH  by  two-digit  WUC 

E.  Break  rates 

F.  Fix  rates 

G.  Abort  rates— In-flight  and  before-flight 

2.  For  all  F-15s,  worldwide,  and  for  WUC  74  for  first  quarter  1993: 

-Overall  MTBMA 

-Listing  of  all  LRUs,  by  serial  number,  with  more  than  three  maintenance  actions  in  first  quarter  1993 
-Listing  of  all  LRUs,  by  serial  number,  with  MTBMA  less  than  overall  group  average 

3.  For  F-16,  part  number  1829003,  for  May  92  through  April  93,  list 

Serial  number.  WUC  (14DPO  and  140QO).  tail  number,  total  sorties,  total  maintenance  actions,  total 
maintenance  man-hours,  number  of  failures,  number  of  removals,  number  of  CNDs,  number  of  repairs, 
number  of  NRTSs,  narratives. 

4.  Algcffithms  in  REMIS  and  REMISTALK  ior  calculation  of  MC  rates,  MTBR,  MTBCF. 

Example  run  comparing  variables  drawn  fiom  REMIS  and  REMISTALK  to  see  whether  or  not  they  are 
the  same. 


Figure  IV-1.  First  Data  Request 


CAMS/REMIS  Data  Request — 26  May  1993 

1.  List  of  current  TCTOs  for  B-1  fiom  GCSAS 

2.  Seymour  Johnson  LANTIRN  and  backshop  automatic  test  equipment  status 

3.  Fot  December  1992,  F-16C/D  at  ACC: 

Mission-oqtable  rates 

Flying  hours 
Sorties 

Maintenance  man-bouis  per  flying  hour 

Mean  time  between  maintenance  actions— Type  1.  Type  2,  Type  6 

Total  number  of  aircraft 

Total  man-lKNus 

Total  failures 

4.  Approved  and  actual  ccmfigutation  for  any  tail  number  of  the  B-1 

5.  From  CAMS,  number  of  transactioas,  by  month,  January  1991  to  latest  available,  for  4tb  Wing  at 
Seymour  Johnson  AFB  and  388tb  Fighter  Wing  at  Hill  AFB 


Figurs  iV>2.  Sscond  Data  Raquast 
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V.  EVALUATION  OF  EXISTING  SYSTEMS 


This  chapter  provides  a  comprehensive  view  of  the  functions  of  the  three 
information  systems.  The  topics  covered  are  as  follows: 

•  Functionality — by  major  category  (inventory,  status,  utilization,  scheduling, 
debrief.  Maintenance  Operations  Center  (MOC),  maintenance  reporting  and 
scheduling,  supply  system  interface,  engine  management,  cannibalization 
management,  configuration  management.  Time  Compliance  Technical  Order 
(TCTO)  management,  personnel  training/availability/scheduling,  shop 
production  planning,  mobilization  planning,  and  deployment),  and  by  user 
group  (base,  depot,  MAJCOM,  contractor). 

•  Scope — number  (and  location)  of  weapon  systems  and  other  equipment 

•  Operating  characteristics — system  performance  and  technology  (including 
availability,  response  time,  system  architecture,  software,  and  operational 
management  and  control),  ease  of  use  (including  observations/discussions  with 
users  and  the  formal  survey  taken  at  the  Operational  Assessment  at  Seymour 
Johnson  AFB). 

•  Data  accuracy  and  completeness — including  Recovered  Functionality 
(RECFU)  11,1  the  accuracy  of  engine  data,  information  drawn  from  the  two- 
level  maintenance  experiment,  analysis  of  data  developed  during  phr  >e 
inspections,  other  information  on  data  loss,  and  the  integrity  and  security  of 
data  input. 

•  Adaptability — including  our  vision  of  the  future  Air  Force  organization, 
management,  and  technology  as  it  bears  on  maintenance  information  systems. 
This  section  addresses  the  capabilities  of  the  information  systems  to 
accommodate  new  technologies,  and  the  capabilities  of  the  systems  to  move 
toward  the  evolving  requirements  of  the  future  Air  Force. 

•  Logistics  and  operational  effectiveness — the  information  systems’  affect  on 
logistics  measures  of  effectiveness  and  operational  capability. 

The  chapter  is  necessarily  long  because  it  includes  detailed  information  about  the 
^sterns.  It  identifies  system-specific  deficiencies  and  provides  the  foundation  for  defining 
the  alternatives,  described  in  Chapter  VI,  and  their  costs,  presented  in  Chapter  Vn. 


1  RECFU  n  is  a  REMIS  initiative  to  recover  functionality  that  bad  been  delayed  due  to  funding  cuts. 
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A.  FUNCTIONALITY 


1 .  Background 

The  functionality  of  CAMS,  REMIS,  and  TICARRS  refers  to  the  types  of  tasks  and 
actions  that  these  information  systems  support  during  maintenance  planning,  scheduling, 
and  performance.  For  example,  the  information  system  keeps  track  of  the  inventory  of 
aircraft  held  by  the  Air  Force,  along  with  the  record  of  the  organizational  units  to  which  the 
aircraft  arc  assigned  and  the  units  that  actually  have  possession  of  them.  The  system  also 
records  the  transactions  dealing  with  an  aircraft,  including  its  original  acquisition,  its 
shifting  among  various  organizational  units  and  bases,  its  movement  into  depot,  and 
ultimately  its  retirement  In  addition  the  system  records  modifications  and  repair  actions  that 
have  occurred  on  the  aircraft.  This  function  includes  a  large  number  of  supporting  and 
derivative  system  activities.  Comparing  the  functions  of  CAMS,  REMIS,  and  TICARRS  is 
difficult  because: 

•  there  is  no  promulgated  standard  set  of  functions  that  the  information  system 
must  perform, 

•  even  where  there  is  full  agreement  on  a  particular  general  function,  the  exact 
manner  in  which  a  specific  system  might  carry  out  that  function  is  not 
standardized,  and 

•  each  system  has  its  own  genesis,  based  on  different  perceived  needs  or 
requirements,  and  was  designed  and  developed  by  different  personnel  and 
orgaruzations.  The  systems  are  organized  in  diffnent  ways  and  use  different 
traminology. 

Nonetheless,  it  is  critical  to  this  assessment  to  identify  and  evaluate  the  range  of 
functions  that  each  of  the  information  systems  performs.  It  is  also  important  to  address  the 
different  sub>functions  within  a  particular  function  because  this  affects  the  amount  of 
software  that  must  be  generated  to  give  the  information  systems,  CAMS/REMIS  and 
TICARRS,  equivalent  functitmality. 

2 .  General  Assessment 

In  order  to  assess  the  differences  in  functionality  between  the  systems,  we 
examined  detailed  information  on  the  tasks  the  systems  are  supposed  to  perforau  the  taskis 
they  actually  perform,  and  the  various  points  at  which  they  diverge.  As  an  overview,  TaUe 
V-1  lists  16  principal  functional  areas  and  some  of  the  more  important  sub-functions 
included  in  the  major  areas.  It  also  provides  our  assessment  of  which  functions  each 
informatkm  system  includes. 
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Table  V-l.  System 

Function  Comparison 

Functions  Compared 

CAMS* 

REMIS 

TICARRS 

a. 

iEqiuponent  ]n«ent(My 

Identification 

X 

X 

X 

Location 

X 

X 

X 

Assigned/Possessed 

X 

X 

X 

Transactions 

X 

X 

X 

History — base  level 

X 

X 

X 

History — fleet 

X 

X 

b. 

£^uii»neni,Statas 

Cunent— base 

X 

X 

X 

Current — fleet 

X 

X 

History — base 

X 

X 

X 

History — fleet 

X 

X 

c. 

S^IOipiiDCOt.  UdUlZtiJDO 

Current— base 

X 

X 

X 

Cunent — fleet 

X 

X 

History — base 

X 

X 

X 

Histny — ^flect 

X 

X 

& 

c, 

i. 

'FltllbtSch^iilb;  ' 

’St^pportMOC  '  '  '  '  '  ? 

Dcbdefine  ,  ^  , 

.  >  X 

y 

Flight  profile,  data,  discrqnncies 

X 

X 

Woric  order  generation 

X 

X 

History— base 

X 

Narrative  not 
now  retrievable 

X 

Histcnry — ^fleet 

Narrative  not 
now  retrievable 

X 

mmmrnmsm; 

Worit-otder  generation  X 

Maintenance  planning  and  scheduling  X 

Job  tracking  X 

X 

X 

Tline  Change  Item  Scheduliiig/Tracldng 

Phase  inspection 

X 

X 

X 

X 

Failure  histories 

X 

Narrative  not 
tiow  retrievable 

X 

Corrective  action  histories 

X 

Narrative  not 
nowretrievaUe 

X 

Reliability  measurement 

DiilicuUto 

X 

retrieve  part- 
number  level 

MaintaiiutbUi^  measurement 

Difficult  to 

X 

retrieve  at  part- 
number  level 

Serialized  pan  inaintenance  history 
Qmirdl/Qiialiiy  Assurance 


X 

X 


Not  observed 
X 


X 

X 


Table  V-1.  System  Function 

Comparison 

(Continued) 

Functions  Compared 

CAMSa 

REMIS 

TICARRS 

h. 

Matotenance-S  apply  System  Interface 

Job  site  parts  ordering 

X 

Interface  with  configuration  file 

X 

Track  status  of  parts  order 

X 

Mission  capable,  awaiting  parts  management 

X 

i 

Comprehensive  Engine  Managem»it 

X 

j* 

CanafbaUzadons  Trackiog/Managwnoat 

Incomplete 

k. 

Ct»ngitratIon  Traddng/Managemettt 

.  ,X  .  .  , 

L 

TClOMac^fimeot 

iiiiiliiiii 

Sitelii 

FersormdTsainlng/Availiibiltty/Sciiechtlu^ 

X 

n. 

Shc^  ProducUon  Pianntng/Scbedaling/Ccmtrol 

X 

0, 

Mobifrzadcm  PlaAnmg 

P- 

System  Deployability 

Eieployable  communications  links 

X 

X 

IDeployable  inframation  system 

X 

Note:  The  letters  to  the  left  of  the  fuDctions  correspond  to  the  letters  of  the  subsections  that  describe  them. 
■  CAMS  functions  are  performed  only  at  the  base  level, 
b  Not  fielded;  in  testing. 


Like  any  classification  scheme,  the  structure  of  the  list  in  Table  V-1  has  some 
arbitrary  features.  Some  tasks,  such  as  TCTO  management,  are  considered  major  functions 
because  of  the  emphasis  placed  upon  them  in  the  maintenance  community  or  the  amount  of 
activity  they  involve.  Regardless,  the  list  serves  as  a  convenient  framework  for  organizing 
the  comparison  of  system  functionality  that  has  resulted  from  our  field  trips,  interviews, 
review  of  formal  documentation,  and  observations  of  the  TICARRS  92  Operational 
Assessment  held  at  Seymour  Johnson  Air  Force  Base. 

Table  V-1  shows  that  neither  CAMS/REMIS  nor  TICARRS  has  provisions  for 
mobilization  planning.  TICARRS  does  not  provide  any  of  the  sub-functions  of  a 
maintenance-supply  system  interface,  nor  does  it  provide  a  Comprehensive  Engine 
Management  System,  a  personnel  training/availability/scheduling  subsystem  or  a  module 
for  shop  production  planning/scheduling/control.  CAMS/REMIS  does  not  now  provide  a 
deployable  system,  though  it  can  support  deployed  operations  through  communications 
links. 

Of  course,  the  simple  assessment  in  Table  V-1  gives  no  indication  of  how 
completely  or  how  well  the  systems  perform  in  any  of  the  major  functional  areas.  The 
speed  and  facility  with  which  the  systems  perform  the  functions  are  assessed  later  in  the 
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report.  This  section  assesses  the  completeness  of  each  system  and  the  approach  of  each  to 
the  functions. 

3 .  Specific  Assessment 

This  assessment  focuses  on  the  relative  strengths  and  weaknesses  of  each  system 
with  respect  to  the  various  functions.  To  the  extent  that  some  one  or  other  organizational 
level  has  experienced  or  is  concerned  about  the  identified  strength  or  weakness,  that 
information  is  also  included. 

It  is  difficult  to  separate  clearly  the  presence  of  a  function  in  a  system  from  the 
quality  of  the  system’s  performance  of  that  function.  That  is  why  the  comparison  shown  in 
Table  V-1  cannot  fully  describe  the  essential  differences  between  CAMS/REMIS  and 
nCARRS.  Poor  performance  of  a  function  may  be  as  bad  or  worse  than  not  ha>  ing  the 
function  at  all,  and  differences  in  the  performance  of  one  function  may  compensate  for 
difierences  in  the  performance  of  others. 

a.  Equipment  Inventory 

CAMS  and  TICARRS  have  the  capability  for  direct  entry  of  equipment 
identification,  its  physical  location,  the  organization  to  which  it  is  assigned,  and  the 
organization  that  has  actual  possession.  They  also  can  enter  transactions  recording 
equipment  gains  and  losses  along  with  the  information  associated  with  the  equipment  being 
transferred.  Because  CAMS  is  a  distributed  data  base,  the  records  must  be  physically 
shifted  when  equipment  is  transfened  from  one  CAMS  base  to  another.  The  central  data 
base  feature  of  TICARRS  fulfills  this  function  by  recording  the  change  in  ownership  or 
possession,  and  other  information  does  not  need  to  be  shifted  because  it  is  already  available 
to  anyone  on  the  system.  In  the  current  versions  of  its  conversations  and  screens, 
TICARRS  identifies  the  organizational  unit  and  base  to  which  the  aircraft  is  assigned  but 
not  the  command  or  the  program  element  of  the  Future  Years  Defense  Program.  Given  the 
structure  of  the  TICARRS  database,  adding  identifier  fields  for  the  latter  should  be  a 
relatively  simple  software  maintenance  task. 

Part  of  the  transfer  process  consists  of  scheduling  and  recording  the  29  actions 
required  for  a  transfer  inspection.  Accomplishing  this  is  more  cumbersome  in  TICARRS 
than  in  CAMS. 

REMIS  receives  inventory  records  and  transactions  from  CAMS.  With  the 
completion  of  RECFU  H,  this  updating  is  to  occur  every  hour  and  the  discrepancies 
between  CAMS  and  REMIS  data  over  which  earlier  concerns  were  expressed  are  to  be 
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reconciled.  The  MAJCOMs  and  Air  Staff  are  the  principal  users  of  these  REMIS  data  for 
planning  and  directing  operations. 

b.  Equipment  Status 

This  function  records  the  availability  of  the  equipment  and  its  capacity  (or 
incapacity)  to  carry  out  its  assigned  mission(s),  along  with  the  number  of  hours  the 
equipment  has  been  in  various  conditions.  During  the  Operational  Assessment  of 
TICARRS  92  at  Seymour  Johnson  Air  Force  Base  (AFB),  backshops  were  favorably 
impressed  with  TICARRS’s  ability  to  keep  track  of  the  status  of  the  automatic  test 
equipment  (ATE)  that  is  essential  to  repairing  line  replaceable  units  (LRUs).  CAMS 
provides  a  similar  function  in  its  Automatic  Equipment  Reporting  System  based  on  the 
TICARRS  approach,  but  it  has  not  received  wide-spread  use.  Also,  the  MOC  personnel 
liked  TICARRS’s  ability  to  give  notice  automatically  when  aircraft  become  “hangar 
queens”  as  their  downtime  accumulates.  The  quality  of  the  REMIS  data  was  generally 
thought  to  be  equal  to  that  of  TICARRS  data,  but  at  one  air  base  there  was  some  concern 
about  REMIS  equipment  status  information. 

As  in  the  case  of  the  inventory  data,  the  MAJCOMs  are  principal  users  of  the  status 
information.  In  at  least  one  instance,  MAJCOM  personnel  felt  that  the  edit  and  error 
correction  procedures  for  the  EIMSURS  module  of  REMIS  was  causing  a  degradation  in 
the  timeliness  and  quality  of  its  status  information.  Edits  pushed  back  to  the  originator  for 
verification  or  correction  are  retained  indefinitely  or  until  corrected.  Those  not  corrected 
quickly  are  sent  for  review  by  the  supervisor  of  the  person  who  made  the  original  entry. 

c.  Equipment  Utilization 

This  function  records  the  elapsed  time  of  operation  of  each  weapon  system  or  piece 
of  equipment.  It  may  also  make  provision  for  recording  other  measures  of  use  of  the 
equipment  such  as  sorties,  touch-and-go  landings,  full  landings,  engine  cycles,  or  number 
of  tests  performed.  These  data  are  entered  into  the  information  system  by  flight-line,  staff, 
or  shop  personnel.  CAMS  receives  directly  from  the  Air  Staff  data  on  flying  time 
allocations  to  each  unit  and  reports  comparisons  of  allocated  versus  actuals.  Principal  users 
of  these  data  include  the  Air  Staff,  MAJCOMs,  engine  management,  and  plans  and 
scheduling  personnel.  The  latter  two  use  the  data  for  plarming  the  inspections  that  are  based 
on  utilization  rates.  TICARRS  does  not  have  an  interface  with  the  K()02  system  through 
which  the  Air  Staff  allocates  and  monitors  flying  hours.  It  will  need  to  develop  such  a 
crqrability. 


At  Seymour  Johnson,  there  was  a  concern  that  engine  utilization  in  TICARRS  is 
measured  only  in  hours  and  not  in  the  other  measures  (cycles,  afterburner,  etc.)  that  are 
important  to  engine  management.  Squadron  and  wing  staff  had  problems  with  the 
accumulated  flight  hours  being  produced  by  TICARRS  but  could  not  verify  whether  or  not 
this  might  be  due  to  the  original  CAMS  data  that  were  entered  into  the  TICARRS  system 
for  the  assessment.  (Incorrect  cumulative  times  adversely  affect  the  scheduling  of  time 
change  items.)  At  the  Seymour  Johnson  assessment,  it  was  very  inconvenient  to  track 
sortie  histories  over  a  month  since  sorties  could  only  be  retrieved  a  day  at  a  time  with 
TICARRS. 

Air  Combat  Command  (ACC)  personnel  told  the  study  team  that  they  have  been 
trying  to  rely  on  REMIS  data  to  report  MAJCOM-level  flying-hour  information  to  the  Air 
Staff  on  a  monthly  basis.  Data  they  receive  have  been  incomplete  and  inconsistent,  forcing 
them  to  obtain  verbal  reports  from  individual  wings. 

Through  requests  for  data  from  the  CAMS/REMIS  Program  Management  Office 
(PMO)  and  the  F-15  System  Program  Office  (SPO),  the  IDA  study  team  conducted  a 
structured  informal  test  of  the  effectiveness  of  the  two  systems’  functionalities.  A  portion 
of  these  requests  dealt  with  information  on  aircraft  inventories,  status,  and  utilization.  Both 
systems  produced  numerical  information  that  was  within  a  few  percentage  points  of  each 
other  for  these  various  measures.  However,  due  to  difficulties  created  by  the  RECFU  II 
effort,  REMIS  was  not  able  to  “find”  the  data  for  the  full  time  period  for  which  the 
information  was  requested. 

d.  Flight  Scheduling 

Tliis  function  permits  Air  Force  wing  and  squadron  personnel  to  create,  nKxiify, 
update,  and  delete  aircraft  flying  schedules.  User  assessments  of  CAMS/REMIS  and 
TICARRS  performance  of  this  function  did  not  appear  to  be  substantially  different.  At  the 
Operational  Assessment,  the  observation  was  made  that  the  TICARRS  version  takes  more 
time.  Also,  concern  was  expressed  that  TICARRS  docs  not  facilitate  viewing  schedules 
over  longer  periods  as  well  as  CAMS  does. 

e.  Support  of  the  Maintenance  Operations  Center  (MOC) 

This  fiincticm  supplies  the  information  needed  in  the  MOC  to  keep  detailed  track  of 
scheduled,  in-progress,  and  completed  aircraft  flights,  as  well  as  the  condition  of  non¬ 
flying  aircraft  As  Table  V-1  indicates,  both  the  CAMS/REMIS  and  TICARRS  systems 
serve  this  function,  but  there  are  variances  in  their  performances. 


At  Seymour  Johnson,  the  MCXT  personnel  observed  that  the  Automated  MOC 
(AMOC)  portion  of  TICARRS  does  more  than  CAMS  and  with  greater  facility.  AMOC 
automatically  tracks  a  number  of  features,  including  crew  ready  time,  crew  show,  engine 
start,  and  taxi  time.  CAMS  relies  more  on  hard-copy  forms  for  some  of  these  features  and 
does  not  automatically  flag  deviations.  The  AMOC  is  considered  more  user-friendly  than 
CAMS.  At  Langley  AFB,  personnel  believe  the  CAMS  version  has  all  the  necessary 
information  but  places  a  heavier  burden  on  the  MOC  operations. 

f.  Debriefing 

In  this  function,  CAMS  and  TICARRS  automate  the  procedure  of  recording, 
modifying,  updating,  or  deleting  information  about  individual  sorties,  mostly  supplied  by 
the  pilots  flying  the  sorties.  The  data  include  operating  time,  flight  profile  information,  and 
aircraft  system  operating  discrepancies.  To  report  the  latter  of  these,  the  CAMS  version  of 
debriefing  uses  Automated  Aircraft  781-Series  Forms,  a  separate  subsystem  that  serves 
other  CAMS  functions  as  well.  Provision  is  also  made  for  automatic  generation  of 
maintenance  work  orders. 

There  are  few  important  distinguishing  features  between  CAMS  and  TICARRS  in 
this  function.  Debriefing  personnel  were  generally  satisfied  with  either.  At  Seymour 
Johnson,  however,  debrief  personnel  reported  that  CAMS  is  more  difficult  to  learn,  that  the 
CAMS  manuals  are  not  reliable,  and  that  error-correction  with  CAMS  is  more  difficult, 
possibly  involving  the  data  base  manager’s  assistance.  Wing  staff  at  Langley  AFB  think 
that  the  ability  to  retrieve  from  REMIS  narrative  information  transmitted  to  it  from  the 
CAMS  debrief  would  be  extremely  helpful.  However,  TICARRS  must  develop  the 
capability  to  generate  the  Automated  Aircraft  AFTO-781  Series  Forms  if  it  is  to  have 
equivalent  functionality,  especially  with  respect  to  the  strategic  bomber  portion  of  the 
ACC  fleet. 

g.  Maintenance  Reporting  and  Scheduling 

The  maintenance  reporting  and  scheduling  function  deals  with  initiating, 
scheduling,  and  tracking  both  on-equipment  and  off-equipment  repair  work,  both 
scheduled  and  unscheduled.  As  shown  in  Table  V-1,  it  covers  a  wide  range  of  sub¬ 
functions. 

Both  CAMS/REMIS  and  TICARRS  have  modules  that  carry  out  these  sub¬ 
functions  but  their  approaches  differ  significantly  in  some  instances.  TICARRS  is  weapon- 
system-oriented  and  predominantly  takes  the  approach  of  recording  events,  whereas  CAMS 
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is  work-center-  or  event-oriented  and  is  more  proactive  in  supporting  required  maintenance 
activities. 

While  the  features  of  both  the  CAMS  and  TICARRS  work-order  generation  were 
generally  considered  to  be  quite  good,  some  flight-line  personnel  at  the  Seymour  Johnson 
assessment  believed  the  process  to  be  easier  in  TICARRS  because  its  screens  and  error 
messages  better  facilitate  the  data  entry.  However,  it  was  also  observed  that  TICARRS 
makes  a  “remove”  action  without  a  coiresponding  “replace”  action  (a  common  occurrence 
when  configuration  changes  are  being  made  to  an  aircraft  for  a  mission)  more  difficult.  In 
TICARRS,  the  closure  of  such  a  remove  action  must  be  made  by  the  technician’s 
supervisor,  a  requirement  that  TICARRS  advocates  claim  imposes  more  discipline  and 
information  accuracy  on  maintenance  data  entry.  Shop  personnel  also  observed  that  while 
TICARRS  does  permit  one  work  center  to  create  work  orders  for  other  work  centers,  those 
work  orders  do  not  load  the  other  work  centers  under  the  same  job  control  number,  a  major 
advantage  of  CAMS. 

The  observations  regarding  maintenance  planning  and  scheduling  and  job  tracking 
were  mixed.  For  some  sub-functions  people  favored  CAMS/REMIS,  for  others  they 
favored  TICARRS. 

At  Seymour  Johnson,  the  consensus  was  that  scheduling  is  done  more  easily  in 
CAMS  than  in  TICARRS  (TICARRS  focuses  on  maintenance  events  in  terms  of  the 
aircraft,  not  on  the  work  centers  where  the  repairs  are  carried  out).  This  applies  to  job 
tracking  in  the  shops  as  well.  There  was  almost  total  agreement  at  all  levels  of  operations 
and  staff  that  TICARRS  needs  a  display  like  the  CAMS  screen  380,  which  permits  tracking 
of  open  work  orders  by  work  center.  CAMS  also  gives  a  better  snapshot  of  the  work  that 
has  been  performed  on  a  particular  job-control  number  (JCN). 

Another  difficult  problem  for  TICARRS  was  the  required  14-day  records  check, 
which  reviews  the  records  of  inspections,  time  change  items  (TCIs),  open  discrepancies, 
engine  data,  and  so  on,  of  a  specific  aircraft  (for  all  aircraft)  as  of  that  time.  CAMS 
performs  this  automated  records  check  using  four  screens  whereas  TICARRS  cannot 
perform  this  function  directly  from  any  screen.  At  Seymour  Johnson,  it  was  claimed  that 
TICARRS  could  perform  this  through  a  query,  but  the  Assessment  Team  repotted  that  the 
query  did  not  work.  Moreover,  having  to  use  a  query  to  cany  out  this  function  is  more 
laborious  and  time-consuming  than  it  should  be  for  such  a  routine  requiremenL 

Both  CAMS  and  TICARRS  carry  out  TCI  and  inspection  scheduling.  At  Seymour 
Johnson,  production  and  scheduling  personnel  observed  that  CAMS  requires  calculation  of 
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the  TCI  due  dates  off-line  and  manual  entry  through  several  screens,  whereas  TICARRS 
calculates  and  enters  them  automatically  once  installation  of  the  part  on  the  aircraft  is 
recorded  in  the  system  by  its  work  unit  code  (WUC).  Personnel  encountered  difficulties 
using  TICARRS  in  tracking  work  related  to  inspections  because  it  accounts  for  work  by 
part  and  not  by  work  center,  particularly  when  multiple  work  centers  were  involved. 
Finally,  a  minor  rigidity  or  omission  in  TICARRS  was  reported.  While  it  provides  for 
recording  narratives  for  TCIs,  TCTOs,  and  the  “fix”  portion  of  inspections,  it  has  no  such 
provision  for  the  “look”  pan  of  inspections. 

Concern  was  expressed  for  how  well  CAMS  can  track  work  on  those  LRUs  that 
may  be  repaired  in  a  two-level  maintenance  context  (i.e.,  where  the  LRUs  are  removed  and 
replaced  at  the  flight  line  and  moved  directly  to  the  depot)  without  any  work  being  done  at 
the  intermediate  level.  CAMS  immediately  loses  sight  of  the  part  once  it  leaves  the  base 
maintenance  and  supply  systems,  whereas  TICARRS  continues  to  keep  track  of  it  (serially 
tracked  items  only)  because  it  is  always  in  the  TICARRS  central  data  base  that  users  access 
directly.  In  CAMS/REMIS,  REMIS  has  the  fleet-wide  information  needed  to  support  two- 
level  maintenance,  but  base-level  users  do  not  have  access  to  REMIS  directly  through  their 
CAMS  terminals. 

The  functions  involving  failure  and  corrective  action  histories  and  reliability  and 
maintainability  data  are  largely  carried  out  by  REMIS  and  TICARRS.  CAMS  is  oriented  to 
base-level  activity  and  does  not  retain  historical  data  indefinitely.  Consequently,  the  scope 
and  extent  of  its  data  base  are  limited  by  design  for  performing  these  functions.  Within 
REMIS,  the  Product  Performance  Subsystem  performs  this  function,  but  it  is  not  as  well 
received  as  TICARRS.  REMIS  receives  narratives  from  CAMS  but  can  retrieve  them  only 
through  the  less-convenient  means  of  specific  queries  or  its  ad  hoc  language, 
REMISTALK.  At  the  F-15  and  F-16  SPOs,  concern  was  expressed  over  REMIS’s 
inability  to  track  reliability  and  maintainability  (R&M)  information  by  aircraft  block 
number.  They  also  want  assurances  that  REMIS  contains  complete  histories  (without  gaps) 
before  they  have  to  rely  on  it  solely.  Before  the  fielding  of  REMIS,  the  B-IB  contractor 
created  its  own  central  data  base,  gathering  CAMS  input  directly  from  the  various  bases 
where  that  aircraft  is  stationed.  It  apparently  continues  to  do  so  rather  than  rely  on  REMIS. 
Other  contractors  insist  that  they  still  need  the  D056  system  tapes  because  they  are  unable  to 
retrieve  the  data  they  need  from  REMIS. 

Observations  about  TICARRS  are  generally  more  positive.  Personnel  in  the  repair 
shops  liked  the  ability  of  TICARRS  to  give  them  part  or  LRU  histories.  The  F-15  and  F-16 
SPOs  rely  on  fleet-wide  visibility  from  TICARRS  for  histories  and  R&M  data.  The 
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narratives  contained  in  the  histories  provide  a  significant  advantage  over  REMIS.  The  F- 16 
SPO  attributes  a  significant  portion  of  the  credit  for  that  aircraft’s  success  to  the  fleet-wide 
visibility  of  the  data  in  TICARRS  and  its  contribution  to  quick  problem  correction. 
Contractors  would  like  longer  time  series  to  be  available  on-line  in  TICARRS  but  concede 
the  off-line  retrieval  to  be  adequate. 

Members  of  the  maintenance  community  at  the  shop,  depot,  and  SPO  levels  were 
very  positive  about  TICARRS ’s  long,  fleet- wide  serialized  part  maintenance  histories  and 
the  ability  those  histories  give  analysts  to  identify  and  track  bad  actors  (although  there  is 
concern  because  TICARRS ’s  data  presently  is  only  as  good  as  the  CAMS  data  that  feeds 
it).  CAMS  is  unable  to  do  this  because  of  its  individual-base  orientation.  Some  users  have 
been  able  to  use  REMIS  to  track  repair  histories-,  and  the  capability  to  track  bad  actors  has 
been  demonstrated  using  REMIS  information  at  one  depot,  Oklahoma  City  ALC. 

Finally,  the  CAMS  and  TICARRS  Quality  Control/Quality  Assurance  functions 
each  consists  primarily  of  a  Product  Quality  Deficiency  Report  (PQDR)  capability.  This  is  a 
direct-entry  function  that  reports  known  or  suspected  deficiencies  of  equipment,  weapon 
systems,  or  related  components.  Since  TICARRS  has  not  been  on  direct  entry  for  several 
years,  its  function  apparently  has  not  been  modified  to  conform  to  die  latest  technical  orders 
on  PQDRs,  As  a  consequence,  at  the  Seymour  Johnson  Operational  Assessment,  the  4th 
Wing  resorted  to  the  CAMS  PQDR  routines.  The  Assessment  Team  also  observed  that 
TICARRS  does  not  make  provision  for  allowing  the  screening  officer  in  quality  assurance 
to  return  the  PQDR  to  the  originator  for  modi&ration  or  closing — a  capability  CAMS  does 
provide. 

The  results  of  the  IDA  data  request  to  the  CAMS/REMIS  PMO  and  the  F-15  SPO 
reinforce  sonu:  of  the  impressions  and  opinions  gathered  during  field  trips  with  respect  to 
the  systems’  abilities  to  provide  R&M  information.  Both  REMIS  and  TICARRS  produced 
aggregate  information  on  maintenance  man-hours,  mean  times  between  maintenance 
actions,  and  total  failures  in  response  to  the  requests.  For  REMIS,  much  of  this 
information  was  obtained  using  REMISTALK.  However,  when  the  request  was  made  in 
terms  of  a  given  part  number  or  specified  that  the  information  be  reported  by  LRU  serial 
number,  the  REMIS  PMO  either  did  not  supply  the  information,  stated  that  it  did  not  have 
the  capalnlity  to  produce  it,  or  supplied  similar  information  for  the  wrong  dates  and  model 
of  aircraft. 
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h.  Maintenance-Supply  Interface 

This  function  facilitates  the  information  system’s  working  directly  with  the 
Standard  Base  Supply  System  (SBSS)  rather  than  through  a  separate  SBSS  terminal.  As  a 
result,  the  user  is  able  to  order  parts  directly  from  the  job  site  and  work  directly  with  the 
configiu^tion  file  to  confirm  that  ordered  parts  are  authorized  and  available  for  use  on  the 
equipment  to  be  repaired.  The  user  can  track  the  status  of  ordered  pans  and  manage  the 
allocation  of  any  pans  that  are  causing  aircraft  mission  status  problems  or  have  caused 
awaiting  pans  status  in  component  repair. 

TICARRS  has  no  provision  for  such  an  interface,  and  that  is  considered  to  be  a 
major  deficiency.  Its  ability  to  track  the  status  of  ordered  pans  is  limited  to  the  technician’s 
making  separate  status  inquiries  through  the  SBSS  terminal  and  manually  entering  the 
information  into  its  related  screen.  Repair  shop  personnel,  particularly  in  the  Avionics  Shop 
and  the  LANTikN  (Low  Altitude  Navigation  and  Targeting  Infrared  System  for  Night) 
shop,  expressed  a  major  concern  because  job-site-pan-ordering  capability  and  ordered-part- 
status  tracking  are  extremely  important  to  their  opmtions.  Another  aspect  of  their  concern 
was  that  without  the  interface,  there  was  no  edit  check  to  assure  that  proper  parts  numbers 
were  being  used.  Similar  concerns  were  expressed  at  the  depot  level. 

i.  Comprehensive  Engine  Management 

This  function  treats  aircraft  engines  like  an  end  item  rather  than  a  component  on  the 
aircraft  The  function  collects  engine  status  and  engine  component  (q)erational  data  that  are 
essential  to  tracking  and  changing  life-limited  serially  controlled  items  on  the  engine.  The 
function  ties  into  the  Comprehensive  Engine  Management  System  (CEMS)  central  data 
base  at  the  Oklahoma  Qty  ALC  to  permit  fleet-wide  tracking  of  depot  visibility  and  analysis 
of  all  engines  in  the  system. 

As  Table  V-1  shows,  CAMS  supports  CSMS  rqxjrting  and  interfaces  directly  with 
CEMS.  TICARRS  treats  engines  like  other  aircraft  components  or  LRUs  for  purposes  of 
collecting  and  tracking  engine  maintenance  information,  it  does  not  interface  with  CEMS. 
During  the  TICARRS  Operational  Assessment,  wing  operations  continued  to  use  this 
function  in  CAMS  because  of  the  importance  of  the  infcnmation  it  supplies  to  engine 
managemoit  and  maintenance  and  the  relationship  of  the  engine  operations  to  flight  safety. 

j.  Cannibalization  Tracking  and  Management 

When  a  part  is  not  available  in  the  supply  system,  authorization  may  be  given  to 
remove  a  part  from  one  aircraft  to  fill  a  hole  in  another.  The  cannibalization  and  tracking 
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management  function  keeps  track  of  the  receiving  and  donor  aircraft  and  the  status  of  the 
part  on-order.  It  also  helps  item  managers  to  allocate  the  supplies  of  the  part  to  fill  the 
resulting  hole.  Without  this  oversight,  a  sequence  of  cannibalization  actions  for  the  same 
part  could  easily  obscure  the  identity  of  the  aircraft  with  the  original  failure. 

Both  CAMS  and  TICARRS  have  versions  of  this  function.  However  at  Seymour 
Johnson,  line  and  supply  personnel  were  not  as  satisfied  with  the  TICARRS  version. 
TICARRS  did  not  provide  a  full  audit  trail  of  sequential  cannibalization  actions  as  a  part  is 
moved  across  several  aircraft.  Each  cannibalization  action  required  the  opening  of  a  new 
wcnk  order  and  JCN  documentation,  including  the  order  placed  on  the  supply  system 
(partially  the  effect  of  TICARRS  not  having  an  interface  with  the  SBSS).  TICARRS  can 
trace  a  particular  part  through  a  sequence  of  cannibalizatitm  actions  only  by  way  of  a  special 
computer  run,  such  as  a  part  maintenance  history  or  a  specific  query  run  in  background. 
The  CAMS  procedures  to  trace  the  series  of  cannibalization  actions  that  a  particular  pan 
may  have  had  were  reported  to  be  less  cumbersome.  Also  TICARRS  did  not,  at  the  time  of 
the  Operational  Assessment,  suppon  cannibalizations  from  removed  engine  to  installed 
engine.  While  provision  was  made  for  such  an  action  subsequently,  it  was  not  used  in  the 
course  of  the  assessment 

k.  Configuration  Tracking  and  Management 

The  configuration  management  function  consists  of  maintaining  the  list  of  part 
numbers  that  are  authorized  to  be  included  on  an  aircraft  ot  in  a  particular  LRU  of  the 
aircraft,  by  quantity  and  location.  It  updates  the  authorized  list  of  parts  as  changes  are  made 
due  to  modifications.  It  also  keeps  track  of  the  parts  (down  to  the  serial  numbers,  where 
applicable)  that  are  actually  cm  each  aircraft  as  lemove-and-replace  actions  occur. 

Opinion  about  the  configuration  function  of  TICARRS  was  generally  higher  than 
that  of  CAMS.  As  indicated  in  Table  V-1,  the  REMIS  configuration  function  has  not  been 
fielded  but  is  currently  in  testing. 

It  should  be  noted  initially  that  because  it  is  a  distributed  system,  CAMS  only  keeps 
track  of  authorized  and  actual  configuraticxi  for  tire  base  at  wdiich  the  CAMS  user  is  located. 
Consequently,  edits  and  checks  for  consistency  are  not  fleet-wide  in  CAMS  as  they  are  in 
TICARRS.  In  CAMS,  at  any  one  time,  a  particular  serial  number  might  be  credited  against 
two  attest  in  two  different  locations  with  detrimental  effects  on  any  attempts  to  track  the 
perfommee  of  the  specifo  serially  controlled  i^rt 

Apparently,  control  of  the  configuration  data  is  much  looser  in  CAMS.  For 
example,  at  Seynxnir  Johnson,  persoimel  reported  that  a  larger  quantity  of  a  particular  part 


can  appear  to  be  on  a  specific  aircraft  than  is  either  authorized  or  can  physically  be  installed. 
This  caused  problems  when  CAMS  data  were  loaded  into  TICARRS  for  the  Operational 
Assessment,  because  TICARRS  rejected  data  that  its  edits  identified  as  being  inconsistent 
with  the  configuration  of  the  aircraft. 

Flight-line  personnel  reported  that  TICARRS  was  more  rigorous  in  its  treatment  of 
configuration  items  and  allowances  for  deviations,  including  serial  number  discrepancies. 
Shop  personnel  reported  that  they  preferred  the  rigor  required  to  change  configurations  in 
TICARRS.  Also,  they  found  the  fleet-wide  visibility  of  all  serial  numbers  of  a  part  to  be 
helpful  for  troubleshooting  and  repair  actions. 

In  discussions  about  the  REMIS  configuration  function,  the  F- 15  and  F-16  SPOs 
expressed  concern  about  the  CAMS/REMIS  capability  to  deal  with  configurations  by 
aircraft  block  number.  Personnel  at  the  Oklahoma  City  ALC  believed  the  REMIS  capability 
would  be  extremely  important  to  its  effectiveness. 

I.  TCTO  Tracking  and  Management 

This  function  provides  an  on-line  capability  for  creating  and  automatically 
transmitting  Time  Compliance  Technical  Orders.  It  also  updates  the  tasks  to  be 
accomplished  under  a  TCTO,  creates  the  work  orders,  and  tracks  and  monitors  the 
progress  made  against  actions  taken.  In  addition,  it  initiates  orders  for  the  necessary 
paits/components. 

The  REMIS  capability  for  this  function  is  part  of  the  Generic  Configuration  Status 
Accounting  System  (GCSAS)  that,  as  previously  noted,  has  not  yet  been  fielded.  The 
judgments  on  the  CAMS  and  TICARRS  versions  include  pluses  and  minuses  for  both. 

At  the  TICARRS  Operational  Assessment,  flight-line  and  wing  staff  observed  that 
CAMS  facilitates  the  scheduling  of  many  backshops  directly  and  simultaneously  if  work  in 
several  is  required  for  a  single  TCTO.  In  contrast,  TICARRS  must  schedule  work  in  the 
backshops  individually  and  through  Production  and  Scheduling.  Flight-line  personnel  were 
also  concerned  that  TICARRS  TCTO  management  did  not  take  into  account  other 
maintenance  actions.  CAMS,  because  of  its  interface  with  SBSS,  automatically  orders 
TCTO  kits  and  monitors  the  status  of  kits,  a  function  that  TICARRS  does  not  have.  There 
was  some  difficulty  surrounding  just  what  maintenance  actions  would  close  a  TCTO  and 
the  extent  to  which  a  particular  shop’s  completed  actions  remained  open  as  long  as  other 
shops’  actions  were  not  closed. 
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Personnel  at  the  Ogden  ALC  believe  that  TICARRS  has  a  significant  advantage  in 
its  ability  to  track  TCTOs  by  aircraft  block  number,  whereas  CAMS  cannot.  The  F-16  SPO 
considers  the  TICARRS  TCTO  information  to  be  more  reliable. 

In  response  to  its  data  requests  of  REMIS.  the  study  team  did  receive  a  listing  of  the 
TCTOs  that  were  current  for  a  particular  B-1  aircraft.  This  list  was  obtained  from  the 
GCSAS  module  of  REMIS.  which  is  currently  in  testing.  At  the  time,  the  approved 
configuration  for  the  B- 1  was  not  complete  and  the  actual  configuration  had  not  yet  been 
loaded  in  GCSAS,  so  it  was  not  possible  to  demonstrate  those  features. 

m.  Personnel  Training,  Availability,  and  Scheduling 

The  personnel  functional  area  provides  an  on-line  means  of  managing  and 
monitoring  the  training  needs  and  achievements  of  the  maintenance  personnel.  It  also  keeps 
track  of  the  availability  and  skills  of  those  personnel  and  helps  in  scheduling  their  duty 
stations  and  times. 

CAMS  has  a  personnel  availability  module  that  was  fielded  in  the  fall  of  1992. 
TICARRS  has  no  provision  for  this  set  of  functions.  The  importance  of  this  function  was 
confirmed  by  the  fact  that  during  the  TICARRS  92  Operational  Assessment,  the  4th  Wing 
was  granted  permission  to  work  around  this  TICARRS  functional  deficiency  by  continuing 
to  use  the  relevant  CAMS  module.  However,  this  difference  in  functionality  may  be 
temporary.  The  Air  Force  has  a  number  of  initiatives  under  way  that  might  provide  a  new 
approach  to  training  management  systems.^  It  is  possible,  therefore,  that  the  training 
management  functionality  included  in  CAMS  could  be  replaced  by  another  system  and  thus 
place  TICARRS  at  no  disadvantage  relative  to  CAMS  in  this  area.  For  now,  however,  this 
cannot  be  assumed. 


^  One  such  initiative  is  called  Bright  Flag,  currently  under  development/review  by  ACC.  Its  purpose  is 
to  provide  a  more  formal  suucture  to  career  training  and  education,  enhance  training/education,  and 
improve  monitoring  of  uaining  and  education  »;tivities.  It  is  to  be  a  paperless  system  and  follow  an 
individual  throughout  his/her  entire  military  career  (replacing  the  G23  document  now).  Bright  Flag 
evolved  from  a  system  called  Base  Training  System,  which  was  an  Air  Force  Materiel  Command 
initiative,  develop^  by  McDonnell-Douglas. 

Bright  Flag  ^plies  to  all  officers,  enlisted  personnel,  and  civilians.  Bright  Flag  focuses  on  job  skills 
training,  quality  improvement  training,  and  professional  and  continuing  education.  Tbe  intent  of  the 
program  is  to  expand  productivity  while  fmoe  structure  (teclines,  provide  tools  to  support  jKofessional 
and  personal  goals,  and  produce  a  highly  qualified  workfixce.  Portions  of  Bright  Flag  ate  being  field 
tested  now.  Feedback  is  being  received  by  A02  on  the  content  of  the  inogram.  While  it  aiq)ears  that 
ACC  will  imptement  Bright  Flag  concqMs,  it  is  not  clear  whether  the  rest  of  tbe  Air  Force  will  ad(^ 
these  initiatives. 
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n.  Shop  Production  Planning,  Scheduling,  and  Control 

This  function  is  an  on-line  planning  and  scheduling  tool  that  lays  out  the  daily, 
weekly,  and  monthly  production  schedules  for  the  various  work  centers  in  the  repair  shops 
and  transmits  these  schedules  automatically  to  those  work  centers.  Its  focus  is  on  the 
equipment  resources  needed  to  carry  out  the  assigned  workload.  CAMS  has  an  Automated 
Scheduling  Module  which  was  fielded  in  1988  and  works  in  a  separate,  personal  computer 
environment.  TICARRS  does  not  have  a  counterpart;  its  reporting  of  removed  “due-ins”  is 
limited  in  assistance  to  shop  supervision’s  scheduling  of  their  shops’  workload. 

o.  Mobilization  Planning 

The  mobilization  planning  function  consists  of  keeping  track  of  all  the  manpower 
and  equipment  requirements  and  the  steps  required  for  an  effective  deployment  of  the 
fighting  unit  to  a  remote  conflict 

Originally  CAMS  was  to  have  a  module  that  would  interface  with  the  Contingency 
Operation/Mobility  Planning  and  Execution  System.  However,  this  function  was  later 
determined  to  be  not  cost  effective  and,  therefore,  urmecessary.  Manual  procedures,  using 
in  part  the  CAMS  capability  to  identify  personnel  to  a  given  Mobility  Group  Number,  now 
satisfy  Air  Force  requirements. 

During  the  Operational  Assessment,  MOC  personnel  observed  that  TICARRS’s 
automated  version  of  the  MOC  (AMOC)  is  able  to  generate  flow  plans  that  would  facilitate 
planning  for  the  fighter/bomber  (non-mobility)  portion  of  mobilization  planning. 

p.  System  Deployability 

This  function  provides  for  the  deployment  of  manpower  and  equipment  resources 
needed  to  set  up  the  information  system  in  support  of  the  fighting  units  in  theater.  The 
requirements  determination  process  has  been  under  way  for  some  time,  but  for  the  present. 
Air  Force  requirements  for  this  function  have  not  yet  been  formally  defined. 

During  Desert  Shield/Desert  Storm,  CAMS  demonstrated  its  ability  to  deploy  a  suite 
of  communications  hardware  that  maintained  a  connection  to  the  aircraft’s  home  base. 
TICARRS  also  made  provision  to  communicate  data  for  Torrejon-based  F-16s  back  to  its 
CONUS  installations  but  was  not  authorized  to  activate  its  commercial  telecommunications 
link.  On  the  basis  of  its  Remote  Engineering  Data  Acquisition  Program,  the  Smart  Data 
System  (SDS)  fielded  a  stand-alone  system  (operated  on  several  personal  computers)  to 
support  maintenance  data  reporting  functions  for  deployed  aircraft.  This  operation  by  SDS, 
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while  employed  without  a  formal  Air  Force  requirement  for  deployability,  served  the 
purpose  of  maintaining  the  F-117  weapon  system  during  Desert  Shield/Desert  Storm. 

ACC  leadership  believes  that  uncertainty  about  the  continued  availability  of 
communications  links  during  a  conflict  and  the  need  for  quick  access  to  information  tailored 
to  support  ad  hoc  deployments  make  it  necessary  to  be  able  to  deploy  part  of  the 
maintenance  information  system  itself,  not  just  to  deploy  a  communications  link.  ACC  is 
leading  an  effort  to  develop  Air  Force-wide  requirements  for  a  deployable  CAMS  system. 
ACC  believes  that  a  deployment  system  must 

(1)  be  rugged,  portable,  and  available  on  the  eve  of  the  deployment; 

(2)  extract  a  subset  of  data  from  the  full  data  base,  including: 

-  TCI  and  phase  inspection, 

-  status, 

-  aircraft  records, 

-  debrief,  and 

-  Job  Data  Documentation  (JDD); 

(3)  be  able  to  backfill  information  on  new  aircraft; 

(4)  have  a  physical  setup  (buildings,  phone  lines,  etc.),  and; 

(5)  update  the  home  station  (daily,  etc.). 

CAMS  cannot  support  such  deployment  today.  TICARRS,  because  of  the  SDS 
experience  in  the  Gulf  War,  appears  to  have  some  advantage  over  CAMS  in  this  area.  This 
issue  is  discussed  further  in  the  section  of  this  chapter  on  adaptability. 

4 .  Conclusions 

This  review  of  the  functions  included  in  CAMS/REMIS  and  TICARRS  indicates 
that  both  systems  have  their  strengths  and  weaknesses.  CAMS/REMIS  has  less  capability 
to  track  serial-controlled  parts,  to  identify  bad  ^tors,  to  produce  accurate  information  on 
aircraft  configuration,  and  to  provide  adequate  failure  and  corrective  action  histories.  Its 
suj^rt  of  the  MOC  provides  less  functionality  than  does  TICARRS. 

TICARRS  has  serious  holes  in  its  functionality  as  well,  including  its  lack  of  an 
interface  with  SBSS,  an  interface  to  the  CEMS,  a  shop  scheduling  management  module, 
and  a  mote  facile  means  to  perform  the  14-day  (automated)  records  check.  The  chorus  of 
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pleas  during  the  Seymour  Johnson  Operational  Assessment  for  a  TICARRS  counterpart  to 
CAMS  screen  380  is  an  example  of  a  lower-grade  functionality. 

On  balance,  TICARRS  is  entirely  missing  a  larger  number  of  important  elements  of 
functionality  than  is  CAMS  or  REMIS. 

B.  SCOPE 

CAMS/REMIS  and  TICARRS  gather  and  manage  information  on  different  weapon 
systems  and  other  (support)  equipment  CAMS/REMIS  can  gather  and  manage  information 
on  many  kinds  of  aircraft  and  other  products,  including  communications-electronics 
equipment,  automatic  test  equipment,  and  other  ground  equipment.  To  date,  TICARRS 
(and  SDS)  has  been  used  with  only  three  types  of  aircraft,  F-15s,  F-16s,  and  F-117s,  as 
well  as  with  automatic  test  equipment  and  trainers. 

Scope,  the  number  of  different  systems  or  equipment  types  handled  now,  clearly 
influences  the  relative  difficulty  and  cost  of  using  the  systems  to  fulfill  the  information 
requirements  of  the  Air  Force.  The  primary  dimensions  of  comparison  for  system  scope 
aie: 

•  number  of  weapon  systems,  which  includes  number  of  different  aircraft  types 
and  models,  and 

•  other  equipment  supported  by  the  information  system. 

We  also  need  to  be  aware  of  the  number  of  bases  using  the  system,  since  introducing  a  new 
maintenance  information  system  at  a  base  (even  one  that  is  home  to  a  kind  of  aircraft  that 
uses  the  system  elsewheie)  can  be  an  expensive  proposition. 

Regarding  the  number  of  weapon  systems,  TICARRS,  if  chosen  as  the  preferred 
system,  would  need  to  have  the  following  aircraft  designs  added  to  its  data  base  operation: 
F-4,  A-10,  F-111,  B-1,  B-2,  B-52,  KC-10,  KC-135,  T-IA,  T-37,  T-38,  T-39,  T-41, 
T-43,  C-12,  C-20,  C-21,  C-22,  C-23,  C-26,  C-130s,  VC-137s,  E-3,  E-4,  H-1,  H-3, 
H-53,  and  H-60.  Many  of  these  have  more  than  one  series  that  would  have  to  be 
accommodated.  MissUes  and  simulators  would  also  have  to  be  added. 

If  the  weapon  systems  on  G081  (C-5,  C-141,  and  C-17)  are  to  use  the  same 
maintenance  information  system  as  other  aircraft,  they  should  also  be  considered  in  this 
evaluation.  The  cost  estimates  in  Chapter  VII  do  not  include  the  costs  of  converting  these 
aircraft  to  a  new  information  system. 
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The  second  dimension  of  scope  concerns  support  of  other  equipment.  For  example, 
at  base  level,  CAMS  currently  handles  equipment  that  support  the  following  functions  and 
users  that  are  not  included  in  TICARRS: 

•  flying-squadron-level  command  section,  intelligence,  life  support,  and 
operations  support; 

•  operations  support  for  the  wing  (including  safety,  command  post,  micro¬ 
computers,  inspection  support,  programs  and  mobility,  group  commander, 
scheduling,  mission  planning,  information  systems  support,  standardization/ 
evaluation,  supply  liaison,  air  traffic  control  (ATC)  and  AlC  support 
functions,  operations  maintenance  support  supervision,  airfield  management, 
various  weapons  and  flight  command  and  control  functions,  operations 
support  squadron  commander,  load  standardization,  intelligence,  orderly 
room,  weather,  life  support,  weapons,  flying  training,  and  operational  plans); 

•  logistics  support  (including  commander,  orderly  room,  programs  branch, 
plans,  maintenance  university,  training  and  administration,  engineering 
technical  services,  and  any  maintenance  complex  self-help  projects); 

•  equipment  maintenance  support  (including  aerospace  ground  equipment, 
production  control,  combat  munitions,  the  Base-Level  Combat  Ammunition 
System,  trailer  maintenance,  combat  support,  munitions  storage,  transit  alert, 
munitions  inspections,  munitions  supply,  munitions  branch,  orderly  room,  and 
commander); 

•  component  repair  support  (including  orderly  room,  commander,  and  precision 
measurement  equipment);  and 

•  wing-level  organization. 

The  list,  which  includes  communications-electronics  equipment  and  other  kinds  of 
equipment,  is  drawn  in  part  from  a  compilation  of  functions  that  remained  on  CAMS 
during  the  Operational  Assessment  at  Seymour  Johnson.  CAMS/REMIS  support  for 
communications-electronics  equipment  goes  beyond  aircraft  operating  bases,  and  includes 
support  for  space-related  equipment. 

C.  OPERATING  CHARACTERISTICS 

The  effectiveness  of  maintenance  information  systems  extends  beyond  functional 
capability  and  scope.  The  operational  characteristics  arc  a  measure  of  how  well  the  systems 
perform  the  functions  and  capabilities  provided  by  the  system  design  in  an  operational  or 
user  environment.  In  evaluating  the  operating  characteristics  of  CAMS,  REMIS,  and 
TICARRS,  the  study  team  focused  on  three  key  aspects  that  encompass  the  issues 
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identified  by  current  and  prospective  system  users.  They  are:  system  performance  and 
technology,  data  integrity  and  security,  and  ease  of  use. 

1 .  System  Performance  and  Technology 

This  subsection  evaluates  those  operational  characteristics  related  to  system 
performance  and  technology.  In  particular,  availability,  response  time,  system  architecture 
and  software  design,  operational  management  and  control,  and  continuity  of  operation  are 
evaluated  for  each  system. 

a.  Availability 

System  availability  is  a  measure  of  the  time  the  system  is  available  compared  to  that 
scheduled.  This  is  usually  expressed  as  a  ratio  or  percent  of  scheduled  available  time  less 
unscheduled  down-time  relative  to  scheduled  time. 

In  practice  this  measure  relates  to  the  central  computer,  and  does  not  necessarily 
reflect  the  availability  of  the  system  as  seen  by  the  user.  The  availability  of  the  system  to  the 
user  is  determined  by  not  only  the  central  computer,  but  also  by  the  availability  of  the 
communications  links  and  terminal  equipment  being  used.  It  is  further  complicated  if  the 
central  computer  is  being  shared  by  several  applications  [as  with  the  Standard  Base  Level 
Computer  (SBLC)  and  CAMS].  The  user  often  cannot  tell  if  the  computer  failed  or  if  the 
application  software  failed  (although  it  makes  no  difference  to  the  user).  On  the  other  hand, 
if  the  central  computer  is  not  available  due  to  unscheduled  down  time,  it  is  clear  that  all  the 
users  are  affected  and  therefore  it  is  a  useful  starting  point  for  an  assessment;  it  provides  an 
upper  bound  to  what  CAMS,  REMIS,  or  TICARRS  availability  looks  like  to  the  users.  If 
the  central  or  host  computer  experiences  low  availability  over  a  prolonged  period,  the 
continual  interruption  in  service  becomes  the  primary  issue  for  both  managers  and  user, 
and  other  operating  characteristics  are  of  secondary  importance. 

None  of  the  systems  under  study  measures  communication  link  busy  calls  or 
outages,  or  terminal  outages  in  a  way  that  reflects  availability  from  a  user  perspective.  Our 
analysis  was  limited  to  the  data  available,  specifically,  central  computer  system  availability 
data  for  all  three  systems,  and  the  CAMS  system  availability  as  reported  by  the  SBLC  PMO 
at  the  Gunter  Standard  Systems  Center  (SSC).  The  data  on  the  central  computer  system’s 
availability  rqport  the  percentage  of  time  the  central  computer  was  operational  and  available 
for  use.  The  data  on  CAMS  availability  report  the  percentage  of  time  the  CAMS  database 
was  available  for  processing  user  transactions.  As  stated  earlier,  these  data  do  not  take 
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into  consideration  any  equipment  outages  due  to  terminal  failure  or  communications 
equipment  failure. 

•  TICARRS  System  Availability.  TICARRS  is  meant  to  be  available  24  hours 
per  day,  7  days  per  week  for  on-line  users.  System  availability  was  measured 
at  99.3  percent  over  a  5-year  period. 

•  REMIS  System  Availability.  The  REMIS  Operational  Performance  Parameters 
(1 1  Januaiy  1993)  set  objectives  of  100  percent  availability,  24  hours  per  day, 
7  days  per  week  as  the  production  baseline.  REMIS  demonstrated  99.82 
percent  availability  over  a  180-day  operational  assessment  period.  In  addition, 
the  availability  rate  of  the  five  REMIS  remote  node  processors  was  99.8 
percent  over  the  same  period. 

•  SBLC  System  Availability.  The  SBLC  operational  performance  requirements 
are  24  hours  per  day,  7  days  per  week  for  on  line  users,  with  an  availability 
objective  of  95  percent.  The  SBLC  PMO  at  Gunter  SSC  provided  data  for  the 
months  of  May  and  June  1993,  which  reported  the  up-time  of  many  of  the 
base-level  SBLC  systems.  (Data  were  not  available  from  all  bases.)  The  uptime 
reported  by  bases  ranged  from  97  percent  to  100  percent,  many  reporting 
uptime  of  99  percent  or  better.  The  overall  24-hour  average  was  99.36  percent 
for  the  month  of  May  for  45  reporting  bases  and  99.06  percent  for  37  reporting 
bases  for  the  month  of  June. 

•  CAMS  System  Availability.  The  CAMS  Operational  Performance  requirements 
are  24  hours  per  day,  7  days  per  week  for  on-line  users,  with  an  availability 
objective  of  95  percent ,  the  same  as  the  SBLC  systems.  Figure  V-1  is  a  graph 
of  the  availability  for  all  of  the  CAMS  systems  associated  with  active  Air  Force 
flying  units  for  the  period  15  February  through  15  May  15  1993.  The  graph 
shows  the  availability  of  each  system  in  terms  of  deviation  over  or  under  the 
goal  of  95  percent  availability. 

The  TICARRS,  REMIS,  and  SBLC  computer  systems  are  operating  at  the  top  end 
of  the  practical  realm  of  availability  given  the  equipment  configurations  and  the  nature  of 
the  applications. 

CAMS,  on  the  other  hand,  suffers  from  an  apparently  wide  variance  in  availability 
from  base  to  base.  The  relatively  poor  showing  of  many  of  the  CAMS  systems  may  be 
attributable  to  the  differences  in  automatic  data  processing  (ADP)  support  and  system 
management  skills  between  bases.  It  is  apparent  that  the  availability  of  CAMS  is  not 
attributable  to  unavailabiliQr  of  the  SBLC  computer  system. 

CAMS  system  availability  is  sometimes  influenced  by  operational  decisions  made  at 
the  base  level.  Some  bases  routinely  operate  the  CAMS  system  for  less  than  24  hours  a 
day.  Virtually  none  of  the  CAMS  systems  is  available  seven  days  per  week.  The  data  in 
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Figure  V-1  reflect  only  those  days  the  CAMS  system  was  in  operation,  weekends  are  not 
counted  as  scheduled  days.  In  addition,  appropriate  adjustments  were  made  for  bases  that 
routinely  close  in  the  late  afternoon.  Especially  when  non-operating  days,  weekends,  and 
routine  early  closings  are  factored  out  (as  they  have  been  in  the  figure),  it  is  not  clear  how 
often  the  decision  to  shut  CAMS  down  coincides  with  the  needs  of  maintenance  personnel. 
Sometimes  it  may,  but  discussions  with  wing-level  personnel  at  several  locations  indicate 
that  CAMS  is  often  taken  down  routinely  at  times  that  are  inconvenient  for  maintainers  who 
have  to  work  at  night  in  order  to  prepare  aircraft  to  fly  in  the  morning.  This  happens  when 
data  base  managers  or  administrators  purposely  shut  CAMS  down  in  order  to  do 
maintenance  on  the  data  base. 


70  J- 

CAMS  Repotting  Bases 

Figure  V*1.  CAMS  Availability  on  SBLC  Systems 

Another  reason  for  the  variation  in  availability  may  be  due  to  errors  in  the  CAMS 
software  that  create  errors  in  the  data  base.  When  this  happens,  it  is  sometimes  necessary  to 
stop  the  operation  of  CAMS  and  repair  the  data  base.  The  extent  to  which  any  of  the  causes 
suggested  are  major  problems  is  speculative  without  more  definitive  data.  Nevertheless,  the 
impact  on  the  users  is  the  same  regardless  of  the  reason  (aside  from  decisions  to  turn 
CAMS  off  at  times  when  maintenantte  activity  is  not  taking  place).  If  the  system  is  not 
available  when  they  need  it,  their  ability  to  do  their  job  is  impaired  and  data  may  be  lost 
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b.  Response  Time 

System  response  time  is  a  measure  of  the  information  system’s  capacity  to  handle 
the  required  input  and  produce  the  required  output  at  a  rate  appropriate  for  the  users.  The 
factors  affecting  response  time  are  many:  power  and  speed  of  the  computer,  design  and 
efficiency  of  the  software,  speed  of  communication  links,  type  of  terminal  equipment, 
amount  of  workload,  type  of  applications,  and  type  of  operational  management  play 
important  roles  in  determining  the  responsiveness  to  user  input. 

For  the  purposes  of  evaluation,  our  analysis  focused  on  the  response  times  of  two 
generic  types  of  input  and  output,  or  transactions.  The  first  is  a  relatively  simple  type  of 
transaction  (e.g.,  a  record  update  or  a  simple  data  query).  This  transaction  typically 
involves  entering  data  at  the  terminal,  depressing  a  transmit  key,  and  transmitting  the  data 
to  the  computer,  which  acts  upon  the  data  by  retrieving  a  record,  updating  it,  and  sending  a 
copy  of  the  updated  record  back  to  the  terminal  operator.  A  simple  query  involves 
essentially  the  same  process,  except  the  computer  would  omit  the  update  activity  and  send 
back  one  or  two  screens  of  report  data. 

The  second  type  of  input  and  output  transaction  is  procedurally  the  same  as  that 
described  above  for  a  simple  query,  except  that  it  is  generally  more  complex  and  the  output 
report  contains  significantly  mote  data  and  is  frequently  sent  to  a  printer  rather  than  the 
terminal  display  for  examination  by  the  requester. 

The  measurement  of  the  response  time  starts  when  the  terminal  operator  depresses 
the  transmit  key  and  ends  when  the  operator  receives  a  response  to  the  transaction  at  the 
terminal.  The  response  time  for  each  of  the  transaction  types  described  above  can  be 
decomposed  into  two  segments,  data  transmission  and  translation  time  and  central 
computer  processing  time. 

Unfortunately,  the  mechanisms  used  to  capture  response  time  data  often  do  not 
include  the  data  transmission  and  translation  times.  This  is  indeed  unfortunate,  since 
response  time  problems  are  frequently  caused  by  an  overloaded  Data  Communication 
Processor  (DCP)  or  overloaded  data  communication  lines.  Discussions  with  technical 
persormel  at  each  of  the  development  sites  provided  insight  into  typical  data  transmission 
time.  The  data  transmission  time  incurred  at  the  base  level  is  about  2  seconds.  However, 
this  can  vary  significantly  if  the  communication  equipment  is  overloaded. 

The  response  time  data  provided  for  the  three  systems  under  study  address  the 
response  time  for  a  transaction  at  the  central  computer.  To  get  an  idea  of  the  response  time 
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seen  by  the  user  sitting  at  a  terminal,  about  2  seconds  should  be  added  to  the  average 
response  time  measured  at  the  central  computer. 

(1)  TICARRS.  Since  TICARRS  has  been  limited  to  accepting  maintenance  data 
via  CAMS  since  1989,  the  study  team  took  advantage  of  the  TICARRS  92  Operational 
Assessment.  During  the  assessment,  the  number  of  daily  transactions  was  recorded  along 
with  the  central  computer  processing  time  for  each  transaction.  Table  V-2  shows  the 
number  of  weekly  transactions  over  the  six  weeks  of  the  assessment,  the  average  number 
of  daily  transactions  over  a  24-hour  period  (5  days  per  week),  and  the  percentage  of  the 
daily  transactions  with  response  times  less  than  3  seconds. 


Table  V-2.  TICARRS  Performance  Data 


Week 

Number 

Transactions 
per  Week 

Transactions 
per  Day 

Percentage  Under 

3  Seconds 

1 

139,077 

27,815 

94 

2 

196,114 

39,223 

97 

3 

197,509 

39,502 

99 

4 

203,302 

40,660 

99 

5 

207,823 

41,565 

99 

6 

295,704 

59,141 

99 

As  indicated  by  the  data,  during  the  first  week  of  the  assessment  6  percent  of  the 
transactions  took  more  than  3  seconds  of  processing  at  the  central  computer.  In  addition, 
the  DCP  at  Seymour  Johnson  was  overloaded  and  contributed  to  relatively  slower  response 
times  during  the  first  two  weeks  of  the  assessment.  Adjustments  made  to  the  DCP  resulted 
in  reported  response  times  of  3  seconds  or  less  for  99  percent  of  the  ttansactions  during  the 
rest  of  the  period.  If  we  assume  a  data  transmission  contribution  of  about  2  seconds  for 
each  transaction.  99  percent  of  the  TICARRS  transactions  over  this  period  would  be  less 
than  5  seconds. 

The  response  time  for  the  second  type  of  transaction  (more  complex  and  lengthy 
output)  was  not  measured;  however,  members  of  the  IDA  study  team  observed  the 
response  time  of  these  type  transactions  to  be  no  more  than  several  minutes  from  the  time 
of  request  to  the  start  of  the  report  generation  at  the  requester’s  printer.  Considering  the 
complexity  of  the  queries  and  the  quantity  of  the  data  processed,  the  response  time  for 
complex  queries  was  acceptable. 

Study  team  observations  indicated  that  the  response  time  for  standard  reports  were 
no  more  than  25  to  30  minutes  for  a  lengthy  report. 


V-24 


During  the  six-week  assessment,  the  TICARRS  system  was  sharing  the  central 
computer  with  other  activities  and  users.  The  Seymour  Johnson  transactions  accounted  for 
an  average  of  2.64  percent  of  the  computer  utilization  for  each  days  operation,  peaking  at 
4.6  percent  during  heavily  used  periods.  Even  though  the  TICARRS  application  was 
sharing  the  central  computer  with  other  applications  and  users,  the  transaction  response 
time  at  Seymour  Johnson  was  relatively  good. 

The  assessment  showed  that  TICARRS  can  provide  adequate  system  response  time 
given  that  the  computer  system  is  not  overloaded  and  processing  tasks  are  properly 
prioritized.  Nevertheless,  it  is  the  view  of  the  IDA  study  team  that  the  computing  power  of 
the  existing  TICARRS  systems  would  need  to  be  substantially  increased  (about  4  times 
more  power)  to  maintain  satisfactory  response  and  throughput  to  support  the  full 
complement  of  Air  Force  weapon  systems  and  functionality  now  provided  by 
CAMS/REMIS. 

(2)  REMIS.  The  REMIS  system  response  time  data  is  based  on  the  operation  of 
REMIS  during  January-May  1993.  At  that  point,  REMIS  was  not  fully  operational.  Two 
of  the  three  major  components  were  available  for  use  throughout  the  Air  Force,  while  the 
third  component  (CCS  AS)  was  undergoing  field  evaluation  and  was  not  to  start  operational 
test  and  evaluation  until  the  third  quarter  of  calendar  1993. 

One  of  the  main  functions  of  REMIS  is  to  collect  and  give  access  to  maintenance 
data  on  a  fleet-wide  basis.  REMIS  must  coUect  data  from  several  different  sources  (CAMS 
base-level  input,  for  example)  and  it  must  distribute  data  to  other  information  systems 
within  the  Air  Force.  It  should  be  remembered  that  the  responsiveness  of  REMIS  may  be 
affected  by  its  need  to  handle  the  workload  imposed  by  these  activities  in  addition  to 
responding  to  user  transactions  and  queries. 

Table  V-3  shows  the  current  (January-May  1993)  system  response  times,  and 
corresponding  computer  utilization  and  report  production  compared  to  the  target  capability 
projected  for  September  1992. 

Examination  of  the  table  reveals  that  while  one  of  the  REMIS  central  computers 
(HQl)  appears  to  be  operating  near  the  maximum  desired  capacity  (60  percent).  The  other 
is  operating  at  only  IS  percent  of  capacity,  demonstrating  a  load-balancing  problem.  The 
average  number  of  daily  users  is  about  95,  who  account  for  an  average  of  226  log-ons  per 
day.  This  is  substantially  lower  than  the  number  of  projected  log-ons,  indicating  there  is 
quite  a  potential  for  growth.  (There  were  2,600  individuals  authorized  access  in  February 
1993). 
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Table  V-3.  REMIS  Performance  Data 


Topic 

Jfanuary-May  1993 
Measurement 

September  1992 
Projection 

HQ1/HQ2  average  computer  utilization 

HQl  =  57% 
HQ2=15% 

HQ1.HQ2  =  68% 

Number  of  standard  reports  per  day 

120 

1,000 

Average  processing  time  per  report 

2.4  bours 

1.2  hours 

Average  number  of  REMISTALK  repeats  per  day 

30 

500 

Average  REMISTALK  processing  time  per  job 

7.5  hours 

8.5  hours 

Average  transaction  (query)  response  time 
(emnputer  processing) 

4seccmds 

4  seconds 

Average  number  of  users  per  day 

y5 

Unknown 

Average  number  of  log-ons  per  day 

226 

800 

Benchmark  measurements  of  REMIS  query  transactions  and  configuration  table 
maintenance  transactions  indicate  that  the  response  time  for  the  simple  transactions  are  in 
the  range  of  3  to  4  seconds  per  transaction  (central  processor  time).  The  more  complex 
query  transactions  were  measured  in  the  range  of  12  to  13  seconds  response  time. 

Average  processing  time  for  a  report  is  higher  than  projected  (averaging  standard 
reports  and  REMISTALK  reports)  and  the  number  of  reports  is  substantially  less  than 
projected  in  September  1992.  The  September  1992  projections  were  based  on  performance 
simulation  runs,  which,  in  turn,  were  calibrated  against  the  actual  REMIS  processing 
activities  over  a  period  of  five  working  days. 

Additional  performance-related  data  is  presented  in  Figure  V-2.  The  data  show  that 
the  standard  reports,  simple  and  complex  transactions,  and  out-bound  data — other,  table 
push-down,  and  system-to-system  output  (SSO) — account  for  less  than  25  percent  of  the 
processing  woikload.  The  other  three-fourths  of  the  workload  is  caused  by  processing  data 
sent  to  REMIS  from  CAMS  via  system-to-system  input  (SSI),  operations  overhead,  and  ad 
hoc  reports  using  REMISTALK. 

This  does  not  appear  to  be  a  satisfactory  situation.  The  system  is  overloaded  and 
unbalanced  with  the  current  workload  without  considering  the  projected  workload.  It  is 
clear  that  improvements  to  performance  are  necessary  for  the  existing  users.  Based  on  the 
data  in  Figure  V-2,  two  functions  (operations  and  REMISTALK)  appear  to  be  candidates 
for  performance  improvements.  Action  was  taken  in  December  1992  to  balance  the 
workload  between  HQl  and  HQ2.  This  resulted  in  a  modest  improvement,  but  the 
workload  has  increased  in  the  meantime.  Continued  efforts  to  balance  the  workload 
between  HQl  and  HQ2  offer  the  greatest  potential  in  the  short  term  to  improve  report 
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throughput  and  response  time.  It  is  critical  that  this  work  be  funded  and  successful  since 
the  SSI  process  and  report  requirements  will  require  even  more  HQl  resources  when 
GCSAS  becomes  operational  across  the  fleet. 


REMISTALK 
29.37% 


Operations 

23.48% 


System-to-System 
Inpi 

23.68% 


Reports 

17.58% 


Systerrvto-System 
Output 

Other  Push-Down 


2.50% 


1.90% 


Figure  V-2.  Utilization  of  REMIS  HQl  Processor  Time 


If  the  September  1992  usage  projections  are  realistic  requirements,  the 
performance-improvement  efforts  outlined  in  the  previous  paragraph  will  not  nearly  be 
sufficient.  The  projections  imply  that  the  system  must  be  capable  of  handling  seven  or  eight 
times  the  cunent  workload.  It  is  not  clear  what  improvements  can  be  made  to  the  REMIS 
configuration  to  enable  the  system  to  support  such  a  workload,  short  of  substantially 
increasing  the  REMIS  system  computing  power. 

(3)  REMISTALK.  REMISTALK  is  a  data  base  query  generator  program  that 
enables  users  to  create  ad  hoc  reports  directly.  REMISTALK  is  intended  to  be  used  for 
those  situations  requiring  data  not  available  from  the  standard  report  programs.  It  provides 
a  mechanism  to  the  everyday  REMIS  user  that  allows  the  user  to  create  ad  hoc  report 
requests  without  waiting  for  programming  services  that  would  otherwise  be  necessary  to 
generate  reports.  REMISTALK  offers  the  prospect  of  creating  a  report  request  and  getting 
the  data  back  in  anywhere  from  six  minutes  to  six  hours,  depending  on  the  amount  of  data 
and  complexity  of  the  report  This  task  takes  from  six  weeks  to  six  months  when  relying 
on  programming  services. 


The  REMISTALK  design  provides  for  a  full  range  of  report-creation  functions, 
including  sorting,  printing,  sort  and  print  break,  if-then  logic,  calculation,  summarizing, 
and  report  formatting.  It  employs  an  easy  to  use  “pick-and-click”  interface  for  report  and 
query  design,  and  provides  users  feedback  at  every  step.  There  is  a  user  library  that  may  be 
used  to  build  or  share  reports  with  other  users,  and  there  is  a  status  board  that  provide 
information  about  the  progress  of  the  report-data-gathering  activity.  The  user  may  save  the 
ad  hoc  report  design  and  reuse  it  or  modify  it  for  different  selection  criteria,  thereby 
reducing  the  effort  to  create  reports. 

REMISTALK  currently  has  two  major  problems.  The  first  problem  stems  from  the 
notion  of  a  Reportable  Database  Area  (RDA).  REMISTALK  is  limited  to  obtaining  data 
from  records  defined  by  the  RDA.  The  reason  for  the  RDA  is  to  eliminate  the  need  for  (and 
ability  of)  users  to  “join”  (a  relational  data  base  operation)  existing  data  tables  into  new 
tables.  The  RDA  provides  pre-defmed  tables  or  views  that  the  users  may  use  for  ad  hoc 
reports.  This  saves  the  users  time  and  effort,  and  reduces  the  amount  of  computer 
resources  required  to  process  the  report.  On  the  other  hand,  if  the  data  a  user  needs  is  not 
included  in  the  RDA,  REMISTALK  cannot  be  used  successfully.  Over  time,  if  funding  is 
made  available,  additional  RDAs  can  be  produced,  which  will  make  more  data  available  to 
more  users.  In  the  meantime,  the  small  number  of  RDAs  (six)  is  the  source  of  a  lot  of  user 
frustration  and  dissatisfaction. 

The  second  problem  with  REMISTALK  is  the  amount  of  time  it  takes  to  provide 
information  in  response  to  a  user’s  request.  The  average  run  time  of  a  REMISTALK 
inquiry  is  roughly  eight  hours.  REMISTALK  has  a  propensity  to  use  a  lot  of  computer 
power  and  thus  it  is  given  a  low  scheduling  priority  within  the  REMIS  central  computer. 
This  low  priority  is  a  contributor  to  slow  user  requests,  but  it  is  not  the  main  factor.  The 
programs  executing  the  REMISTALK  functions  and  obtaining  the  data  use  too  much 
computer  resources,  and  take  too  long  to  execute.  This  problem  needs  to  be  fixed.  We  offer 
the  following  potential  solutions; 

•  Move  the  REMISTALK  operation  to  the  remote  processors  at  the  ALC 
locations.  The  processors  are  under-used  and  the  move  could  provide  relief  to 
the  central  processor  and  provide  faster  service  to  the  REMISTALK  users  by 
allowing  more  computer  power  for  REMISTALK.  Additional  software 
licenses  would  be  necessary  since  REMISTALK  uses  a  proprietary  program 
called  FOCUS. 

•  Artificially  limit  the  amount  of  time  any  ad  hoc  request  may  execute.  This  could 
be  done  by  the  REMISTALK  software  examining  the  request  when  submitted, 
calculating  the  expected  run  time,  and  if  over  the  limit,  inform  the  user  and 
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reject  the  request  Requests  that  take  longer  than  the  limit  could  be  handled  by 
the  REMIS  central  facility  staff. 

•  Create  a  read-only  data  base  that  is  used  exclusively  by  REMIST ALK  and  the 
standard  reports.  This  version  of  the  data  base  would  be  updated  periodically 
(weekly,  monthly)  with  the  data  received  from  CAMS  and  other  sources.  This 
action  would  eliminate  interaction  and  competition  for  resources  with  other 
REMIS  functions  at  the  central  computer  system. 

•  Examine  the  design  and  implementation  of  REMISTALK  for  programming 
defects  and  inefficient  logic  to  reduce  the  amount  of  computer  power  required 
to  execute  REMISTALK. 

•  Balance  the  existing  workload  and  add  another  computer,  or  a  faster,  more 
powerful  computer  at  the  central  system  location.  This  “brute  force”  approach 
would  not  necessarily  fix  the  underlying  problems,  but  it  would  provide 
improved  REMISTALK  report  responses. 

Any  or  all  of  the  above  options  require  funds  that  are  not  currently  in  the  budget  for 
REMIS.  The  REMIS  cost  estimate  prepared  by  the  study  team  includes  funds  for 
examining  all  the  options  above  and  implementing  the  last  two. 

(4)  CAMS.  CAMS  operates  on  the  SBLC  in  a  shared  environment  at  over  100 
bases.  This  evaluation  is  based  on  data  obtained  from  the  SSC  at  Gunter  AFB.  The  IDA 
team  was  given  data  from  96  bases  on  the  average  number  of  daily  transactions  (first  shift) 
and  the  average  transaction  response  times  during  January  and  February  1993  (see  Table 
V-4).  For  the  most  part,  these  transactions  are  simple  queries  and  maintenance  record 
updates 


Table  V-4.  CAMS  Average  Base-Level  Performance  Data 


Tooic 

Value 

Average  number  of  iransacticxis  per  day 

12,188 

Average  response  time 

4.S6  seconds 

Average  number  of  log-rms  per  day 

196 

Percentage  of  SBLC  use  due  to  CAMS 

44% 

The  average  response  time  seen  at  the  terminal  was  6.97  seconds.  Communication 
time  accounted  for  2.41  seconds,  and  the  SBLC  accounted  for  4.S6  seconds. 

Figure  V-3  provides  more  information.  This  graph  shows  the  percentage  of  CAMS 
installations  with  average  response  times  (excluding  communication  time)  within  the  ranges 
given  (1  to  over  8  seconds).  About  30  percent  of  the  CAMS  bases  have  average  response 
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1.00-  2.01-  3.01-  4.01-  5.01-  6.01-  7.00+ 

2.00  3.00  4.00  5.00  6.00  7.00 

Response  Tine  (Seconds) 

Figure  V-3.  Distribution  of  CAMS  Transaction  Response  Times  by  Base 

Figure  V-4  shows  the  number  of  transactions  per  day  per  base,  and  Figure  V-S 
shows  the  response  times  of  the  same  bases.  There  does  not  appear  to  be  any  relationship 
between  the  number  of  transactions  processed  (workload)  and  the  transaction  response 
time.  In  fact,  the  bases  with  a  relatively  low  transaction  workload  are  frequently  the  bases 
with  the  slowest  transaction  response  times.  Also,  some  bases  with  transaction  workloads 
that  are  among  the  highest  show  fast  transaction  response  times. 

While  the  overall  average  of  the  CAMS  transaction  response  times  is  acceptable  by 
Air  Force  standards,  it  is  apparent  from  these  data  that  the  response  times  from  about 
20  percent  of  the  bases  are  a  problem.  This  poor  showing  may  be  due  to  the  differences  in 
ADP  support  and  management  among  the  bases.  Another  possible  reason  for  this  poor 
performance  may  be  that  these  bases  do  not  have  sufficient  computing  power  to  handle  the 
workload.  Problems  with  the  communications  equipment  being  overloaded  or  improperly 
used  would  not  be  reflected  in  this  data  since  the  data  addresses  the  response  of  the  central 
computer  only. 

Whatever  the  reasons,  the  current  plan  to  regionalize  the  SBLC  systems  should 
alleviate  tte  problem. 
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Figure  V-4.  Number  of  Daily  Transactions  per  Base 
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Figure  V-5.  Distribution  of  Base  Transaction  Response  Times 
(Ordered  by  Transactions  per  Day) 
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c.  System  Architecture  and  Software  Design 

This  section  examines  the  way  that  the  architecture  or  structure  of  each  system 
affects  its  operational  characteristics.  Figures  V-6  and  V-7  depict  the  respective  architectural 
configurations  of  CAMS/REMIS  and  TICARRS.  The  CAMS/REMIS  configuration  is 
shown  as  it  will  be  with  the  implementation  of  the  Regional  Processing  Centers  in  1995. 
The  TICARRS  configuration  is  shown  as  it  was  intended  to  operate  (with  direct  data  entry, 
not  entry  via  CAMS). 

Figure  V-6  shows  the  base- level  CAMS  data  bases  residing  in  the  Regional 
Processing  Center  (RPC),  while  data  are  entered  at  the  base  level  via  the  local  base  terminal 
network  and  the  DCP.  Communication  with  the  RPC  from  the  bases  is  via  high-speed 
communication  lines.  CAMS  sends  a  subset  of  the  data  base  information  to  REMIS  via 
Defense  Data  Network  (DDN)  lines  periodically  (some  data  each  hour,  other  data  once  a 
day),  at  which  point  the  data  input  processor  (SSI)  edits  and  translates  the  data  into  REMIS 
format  and  stores  it  in  its  database.  If  the  data  do  not  pass  the  edit  test,  they  are  rejected  and 
returned  to  the  respective  CAMS  base  for  correction.  The  REMIS  data  are  contained  in 
three  modules:  EIMSURS  for  status,  utilization,  inventory,  and  scheduling;  PPS  for 
reliability  and  maintainability  data;  and  GCSAS  for  configuration  data  and  TCTO 
management.  Users  at  the  ALC  and  contractors  access  the  system  for  data  and  reports 
through  one  of  five  remote  processing  systems,  which  perform  all  the  screen  management 
and  user  authorizations.  Weapon  system  configuration  data,  inquiries,  and  requests  for  data 
are  sent  to  the  central  system  from  the  remote  systems,  where  they  are  processed  and 
appropriate  responses  are  returned. 

The  nCARRS  architecture  is  shown  in  Figure  V-7.  Data  are  entered  at  the  base 
level  via  the  base  terminals  and  data  network.  The  DCPs  concentrate  the  data  for 
transmission  over  high-speed  lines  to  the  TICARRS  central  computer.  The  TICARRS 
software  (Conversation  Manager)  determines  the  type  of  transaction  and  establishes  a 
conversation  with  the  users  to  complete  the  transaction.  Data  are  edited  and,  if  acceptable, 
stored  in  the  TICARRS  database.  If  the  data  do  not  pass  the  edit,  the  data-entry  person 
(e.g.,  a  flight-line  technician)  is  notified  on  the  spot  before  the  transaction  is  completed.  If 
the  data-entry  person  disagrees  with  the  edit  results,  the  edit  may  be  over-ridden  with  the 
approval  of  the  supervisor  in  charge  of  the  maintenance  activity.  Other  users,  not  at  the 
base  level,  may  also  obtain  data  and  report  by  establishing  a  link  through  leased  or  dial-up 
lines. 

Table  V-5  summarizes  the  architectural  attributes  of  these  systems. 
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Figure  V-6.  CAMS/REMiS  Configuration 
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Table  V<5.  System  Architecture  Attributes 


Attribute 

TICARRS 

REMIS 

CAMS 

Number  of  systems 

Single  system 

Multiple  systems 

Multiple  systems 

Shaied/dedicated 

Shaied/dedicaied 

Dedicaied 

Sliared 

Data  base  range 

Centralized;  Fleet 

Centralized:  Fleet 

Centralized;  Base 

Data  base  type 

Netwak 

Relational 

Network 

Data  base  (xganization 

Weapon 

Functional 

Organizational 

( 1 )  Number  of  Systems.  The  requirements  for  management  control,  software 
tools,  and  operational  discipline  increases  as  the  numbers  of  systems  and  system  locations 
rise.  If  these  requirements  are  not  met,  the  effectiveness  and  utility  of  the  entire  system 
deteriorates  and  becomes  unsatisfactory  over  time. 

TICARRS  is  a  single  location  system.  There  is  no  requirement  for  inter-system 
management  controls,  tools  or  operational  controls  beyond  that  needed  for  communication 
network  management.  Operational  procedures,  systems’  servicing,  system  availability  or 
performance  problems  are  all  handled  at  one  location  for  one  system.  This  reduces  the 
number  of  operational  and  maintenance  resources  required  to  operate  the  system. 

REMIS  is  a  multiple  location  system  in  that  it  uses  a  six-site  distributed  processing 
system.  The  REMIS  program  development  has  invested  in  a  sophisticated  network  and 
processor  control  center  and  software  tools  to  monitor  and  control  each  of  the  six  systems 
(five  located  at  ALCs).  The  control  center  allows  the  REMIS  operations  staff  to  delect 
problems  at  each  of  the  ALC-located  systems  and  initiate  remedial  action.  This  could  be 
thought  of  as  a  “coupled”  multiple  location  system  due  to  the  control-center  facility. 

CAMS  is  a  multiple-location  system  by  virtue  of  using  the  SBLC  systems  at  each  of 
109  bases  across  the  country.  Operational  control,  servicing  problems,  and  management  of 
the  systems  is  the  responsibility  of  each  base  ADP  center.  Hence,  from  a  CAMS 
perspective,  each  base  must  deal  with  availability,  performance,  and  data  base  problems  in 
its  own  way,  as  local  base  conditions  permit  Therefore,  CAMS  operational  effectiveness  is 
largely  dependent  on  the  ability,  motivation,  and  experience  of  the  base  ADP  personnel, 
and  their  execution  of  base-level  procedures.  There  is  no  coupling  of  CAMS  control  or 
service  in  this  situation.  It  is  no  surprise  to  find  that  the  operational  characteristics  of 
CAMS  vary  significantly  from  base  to  base.  (See  the  previous  subsections  on  system 
availability  and  system  response  time  and  subsequent  subsections  on  ease  of  use  and  data 
accuracy  and  completeness). 
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By  raid- 1994,  CAMS  will  be  operating  at  Regional  Processing  Centers  in  CONUS 
and.  if  plans  are  approved,  in  U.S.  Air  Force  Europe  and  Pacific  Air  Force  as  well.  The 
regionalization  initiative  has  reduced  the  nuraber  of  ADP  personnel  needed  to  ooerale  the 
base-level  systems,  and  will  reduce  the  number  of  different  physical  locations  from  107  (89 
after  base  closures)  to  7  worldwide.  Although  the  number  of  physical  locations  will  be 
reduced,  CAMS  will  still  be  a  multiple-location  system  under  regionalization,  with  a 
separate  data  base  for  each  base.  Some  of  the  operational  control,  servicing,  and 
management  problems  should  improve  with  the  RPC.  However,  the  base-level  CAMS  data 
base  managers  are  a  key  element  in  the  operation  and  management  of  CAMS,  and  they  will 
remain  at  the  bases  to  support  users  on  site.  This  may  leave  CAMS  with  many  of  the  same 
problems  cited  above. 

(2)  Shared/Dedicated.  REMIS  operates  on  a  computer  system  dedicated  to 
providing  weapons  systems  management  information.  TICARRS  shares  a  computer  with 
other  applications,  but  it  would  have  a  dedicated  computer  system  if  it  were  expanded  to 
cover  the  entire  Air  Force.  CAMS  shares  the  SBLC  with  other  applications  at  the  base 
level,  and  this  will  continue  under  regionalization.  While  the  TICARRS  and  REMIS 
operational  organizations  are  able  to  focus  resources  on  the  optimum  operation  and  services 
of  one  application,  the  SBLC  operational  organization  is  (properly)  required  to  divide 
resources  between  all  applications,  and  focus  on  the  optimum  performance  and  services  of 
the  SBLC  system  as  a  whole,  rather  than  solely  on  CAMS.  The  current  evidence  is  that 
may  be  detrimental  to  the  smooth  and  satisfactory  operation  of  CAMS  at  some  locations. 

(3)  Data  Base  Type,  Range,  and  Organization.  These  data  base  attributes 
serve  only  to  provide  a  framework  for  briefly  describing  each  of  the  three  systems’  data 
bases.  They  are  relevant  in  that  database  design  has  a  bearing  on  system  performance,  data 
base  maintenance,  and  growth  ability. 

Both  TICARRS  and  CAMS  data  bases  are  network-hierarchical  types.  REMIS  uses 
a  relational  type  of  data  base.  Network  type  data  bases  are  generally  used  for  high- 
transaction-rate  systems,  they  are  complex  to  design  and  maintain,  and  once  defined,  they 
are  difficult  and  time-consuming  to  change.  Relational  data  bases,  are  not  as  fast,  but  are 
easier  to  maintain,  expand,  and  change,  and  are  somewhat  easier  to  use  in  terms  of 
accessing  and  reporting  different  aspects  of  the  data  base  records.  The  Irey  point  here  is  that 
since  CAMS  uses  the  network  type  data  base,  the  technical  expertise  of  the  data  base 
manager  at  each  base-level  system  must  be  very  high  to  perform  daily  maintenance.  Studies 
by  the  Air  Force  have  shown  that  the  data  base  managers  generaUy  do  not  have  this  level  of 
expertise,  which  may  explain  some  of  the  data  completeness  and  accuracy  problems 
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(discussed  in  a  later  subsection)  and  some  of  the  availability  problems  seen  at  many  of  the 
bases. 

The  TICARRS  and  REMIS  data  bases  are  structured  to  cover  the  entire  (world¬ 
wide)  fleet  of  weapons  systems,  albeit  with  completely  different  data  base  organizations. 
The  CAMS  data  base  is  designed  to  encompass  the  weapons  systems  assigned  to  the  base. 
This  is  reflected  in  the  way  each  of  the  data  bases  are  organized.  The  CAMS  data  base  is 
structured  around  the  base-level  organization  (e.g.  u.iit,  organization,  work  center, 
people/equipment/resource).  The  TICARRS  data  base  is  organized  around  the  weapon 
system  itself.  The  REMIS  data  base  is  functionally  organized  around  three  major  data 
interests  as  served  by  EIMSURS,  PPS,  and  GCSAS. 

The  main  effect  of  this  aspect  of  data  base  design  is  that  both  CAMS  and  REMIS 
must  be  used  to  collect  maintenance  data,  and  provide  a  worldwide  view  of  the  maintenance 
status  and  history.  TICARRS  offers  the  same  capability  in  a  single  centralized  system.  Data 
accuracy  and  completeness  are  affected  by  the  CAMS  data  base  design,  in  that  erroneous 
data  entries  cannot  be  completely  verified  on  a  fleet-wide  basis  without  the  need  to  have 
REMIS  check  the  data.  This  has  resulted  in  use  of  a  series  of  complex  procedures 
involving  REMIS,  CAMS,  CAMS  data  base  managers,  ALC  personnel,  and  base 
maintenance  personnel  to  reduce  the  number  of  erroneous  transactions  generated  at  the 
CAMS  level  and  rejected  at  the  REMIS  level.  When  erroneous  transactions  are  found  at  the 
REMIS  level,  all  following  data  related  to  the  erroneous  record  are  rejected.  This  creates 
gaps  in  the  maintenance  data  that  must  be  filled  by  the  maintenance  personnel  re-entering 
the  data  after  the  erroneous  records  have  been  corrected.  Experience  and  data  measurements 
indicate  very  clearly  that  this  approach  usually  does  not  woric  in  practice. 

Finally,  the  data  base  design  is  frequently  reflected  in  the  way  that  the  data  are 
entered.  One  of  the  most  frequent  complaints  by  users  of  CAMS  is  the  requirement  to  re¬ 
enter  data  when  moving  to  a  new  screen  in  the  same  process.  This  occurs  in  TICARRS  and 
REMIS  as  well.  Requiring  a  user  to  re-enter  data  can  lead  to  errors  in  the  data,  as  well  as 
annoyance  of  the  user. 

d.  Operational  Management  and  Control 

Operational  management  and  control  involves  the  day-to-day  operation  of  the 
computer  center,  making  corrections  to  the  data  base  when  the  contents  are  found  to  be  in 
error,  updating  the  system  with  the  latest  version  or  release  of  the  software,  adjusting  the 
system  scheduling/priority  parameters  to  improve  the  service  to  the  system  users, 
scheduling  regular  preventive  maintenance  time,  and  instituting  and  executing  procedures 
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that  ensure  the  security  and  safety  of  the  system  and  data.  The  operations  personnel  are 
required  to  have  total  knowledge  of  the  system  and  expertise  on  the  powerful  software 
tools  used  to  manage  and  control  the  system. 

The  skills  and  experience  necessary  to  manage  and  implement  operational  control 
are  not  easily  or  quickly  gained.  The  nature  of  the  systems  being  used  to  collect  and 
maintain  the  weapon  systems  data  demands  highly  skilled  and  experienced  operations 
personnel.  All  three  systems,  CAMS,  REMIS,  and  TICARRS  have  essentially  the  same 
requirements  for  operational  personnel  in  terms  of  skill  and  experience.  TICARRS  and 
REMIS  have  the  advantage  of  only  having  to  establish  a  single,  central  staff  of  people  who 
can  focus  full-time  on  the  weapons  maintenance  information  system.  CAMS,  on  the  other 
hand,  works  under  the  control  of  the  operations  personnel  at  each  SBLC  or  RPC,  where 
there  may  be  lesr  emphasis  on  the  weapon  maintenance  information  system  than  there 
would  be  in  a  single  application  environment 

For  example,  the  data  base  administrators  at  the  SBLC  or  RPCs  are  responsible  for 
balancing  the  allocation  of  disk  space  to  satisfy  all  applications,  not  just  CAMS,  thereby 
allowing  for  a  less  than  optimal  situation  for  CAMS.  Another  example  of  how  focus  may 
be  diluted  is  illustrated  in  reports  by  CAMS  development  personnel  that  the  software 
releases  sent  out  with  new  features  and  repairs  to  problems  are  sometimes  not  installed  on 
the  SBLC  systems  for  up  to  10  months  after  they  are  available. 

The  CAMS  data  base  managers  (DBMs)  have  an  important  role  in  the  operation  of 
CAMS.  They  are  not  part  of  the  regionalization  initiative  and  will  remain  at  the  bases.  The 
DBMS  are  responsible  to  support  the  CAMS  users  at  the  base,  trouble  shoot  software 
problems,  and  repair  data  base  problems  that  deal  with  the  logical  structure  of  the  data  base. 
A  survey  conducted  by  the  CAMS  PMO  determined  that  the  majority  of  DBMs  had  less 
than  three  years  experience,  28  percent  had  no  formal  training  and  no  expertise  on  using 
operations  software  tools.  It  is  clear  from  the  systems  availability  data  and  the  system 
response  time  data  that,  since  some  systems  are  operating  well  and  many  are  experiencing 
poor  availability  and  performance,  the  operational  management  of  CAMS  must  be  held 
accountable  for  part  of  the  system  behavior. 

The  effort  to  regionalize  the  SBLC  system  will  reduce  the  need  for  so  many  skilled 
people.  Plans  call  for  consolidation  of  12  to  15  systems  in  each  region.  While  there  is  an 
opportunity  to  improve  the  operational  management  of  the  systems  at  the  regional  centers, 
the  support  burden  on  the  base-level  DBMs  will  not  diminish.  They  will  be  required  to: 

*  {H'ovide  data  base  surveillance. 
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•  fulfill  user  requests  for  background  and  special  reports, 

•  provide  data  base  security, 

•  delete  histories, 

•  provide  minor  data  base  maintenance, 

•  monitor  new  releases  of  CAMS, 

•  coordinate  between  affected  units  on  system  downtime,  and 

•  provide  detailed  system  problem  resolution. 

A  proposal  by  the  CAMS  PMO  to  establish  a  CAMS  data  base  support  function  at 
Gunter  SSC  will  provide  needed  expertise  at  a  central  location  that  can  be  shared  by  all 
bases.  In  addition,  a  CAMS  manager  would  be  established  at  each  base  to  conduct  CAMS 
user  training,  provide  end-user  problem  liaison  and  initiate  user-suggested  CAMS 
improvements.  If  implemented,  this  proposal  could  allow  the  data  base  managers  to  focus 
their  attention  on  the  operations  and  the  resolution  of  problems  that  arise  at  the  base  level 
within  the  current  staffing  levels  (three  to  four  per  base). 

By  comparison,  within  the  TICARRS  central-site  operation,  most  of  these 
functions  are  performed  at  the  central  site,  and  do  not  require  the  same  amount  of  support 
from  site  representatives  as  is  required  with  CAMS.  The  TICARRS  site  representative 
normally  undertakes  the  following: 

•  developing  and  conducting  on-going  end-user  training, 

•  providing  end-user  problem  resolution, 

•  processing  requests  for  functional  enhancements,  and 

•  processing  user  requests  for  background  and  special  reports. 

There  will  be  one  site  representative  per  base  and  two  additional  representatives  per 
base  at  each  Air  Logistics  Center,  consistent  with  historical  experience  with  direct-entry 
TICARRS. 


e.  Continuity  of  Operation 

The  institution  and  execution  of  procedures  that  ensure  the  security  and  safety  of  the 
system  and  data  are  an  extremely  important  aspect  of  operational  management  and  control 
Continuity  of  operation  in  the  event  of  a  disaster  that  prevents  the  operation  of  the  computer 
center  is  of  particular  interest.  The  Air  Force  is  dependent  on  the  weapons  management 
information  system  to  provide  data  regarding  the  readiness  of  aircraft  to  fly.  Therefore,  the 
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weapons  information  management  system  must  have  a  back-up  plan  to  continue  operations 
in  the  event  of  the  destruction  of  the  computer(s)  and  related  equipment 

Each  of  the  systems  being  analyzed  have  alternative  plans  and  back-up  procedures 
that  are  adequate  in  the  event  of  equipment  breakdown,  communication  failures,  power 
failures,  and  other  serious,  but  manageable,  outages.  However,  none  of  these  procedures 
are  of  benefit  if  the  entire  computer  system  becomes  inoperative  as  the  result  of  an 
explosion,  uncontrollable  fire,  and  so  on,  unlikely  as  such  as  event  may  be.  A  back-up  plan 
for  operation  in  case  of  such  an  event  requires  that  the  maintenance  information 
management  system  operate  on  a  compatible  computer  system  in  a  location  remote  from 
and  unaffected  by  the  disaster.  There  must  be  a  capability  to  switch  communication  lines  to 
the  back-up  computer  location  and  there  must  be  a  recent  copy  of  the  data  base,  programs, 
and  operating  procedures  available  to  start  and  run  the  system. 

TICARRS  does  not  have  a  disaster  back-up  capability  in  place.  Because  TICARRS 
is  fed  data  by  CAMS,  a  back-up  capability  is  not  necessary.  CAMS  would  continue  to 
collect  the  data,  and  aircraft  at  the  squadron  level  would  still  have  sufficient  maintenance 
information  allowing  them  to  fly.  Should  TICARRS  become  the  standard  maintenance 
information  management  system,  CAMS  would  no  longer  be  used  to  collect  the 
maintenance  data,  and  a  TICARRS  back-up  capability  would  be  required.  DRC  has 
proposed  a  contract  with  Dataguard  Recovery  Services  to  provide  such  a  capability.  For  a 
monthly  fee.  Data  Recovery  Services  would  provide  TICARRS  with  access  to  an  off-site 
(Loiusville,  Kentucky)  compatible  computer  system  and  communication  facilities  in  the 
event  of  a  disaster  outage  at  the  TICARRS  system  in  Andover,  Massachusetts.  The 
TICARRS  data  base  and  programs  would  be  saved  every  24  hours  (more  frequently  if 
appropriate)  and  a  copy  stored  off-site.  Should  a  disaster  occur,  the  off-site  copy  of  the 
system  and  data  base  would  be  taken  to  Louisville  and  installed  on  the  Bull  system  there. 
The  communication  lines  feeding  TICARRS  at  Andover  would  be  switched  to  the 
Louisville  site,  and  TICARRS  would  operate  there  until  permanent  facilities  could  be 
established.  The  proposed  contract  gives  access  to  TICARRS  for  test  time,  so  that  the 
disaster  procedures  may  be  exercised  in  practice  drills. 

Since  REMIS  is  not  used  to  directly  coUect  maintenance  data  (such  data  is  provided 
by  other  systems  like  CAMS),  a  disastrous  event  knocking  out  the  entire  central  system 
would  not  be  as  critical  as  it  would  be  for  CAMS  (or  TICARRS).  The  REMIS  disaster 
recovery  plan  entails  installing  the  REMIS  data  base  and  programs  (saved  daily  and  stored 
off  site)  at  the  REMIS  satellite  ALC  location  at  Tinker  AFB.  The  computer  system  there  has 
five  processors,  and  would  be  expanded,  as  appropriate,  to  include  sufficient  power  to  at 
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least  collect  data  from  CAMS  and  other  key  maintenance  sources  to  provide  the  Air  Force 
with  a  fleet-wide  view  of  aircraft  inventory  and  status. 

The  disaster  recovery  plan  for  CAMS  is  included  in  the  RPC  disaster  recovery  plan. 
As  with  the  other  two  systems,  the  CAMS  data  base  and  programs  are  saved  daily  and 
stored  off-site.  All  of  the  RPC  installations  are  installed  with  25  percent  more  equipment 
capacity  than  needed  to  service  the  bases  they  support.  In  the  event  of  a  disaster  at  one  of 
the  RPCs  (that  is,  the  entire  RPC  becomes  inoperable),  the  back-up  data  bases  and 
programs  would  be  taken  to  the  remaining  RPCs  and  installed.  The  base-level 
communication  lines  would  be  switched  to  the  RPC  handling  their  workload.  Satellite 
communications  would  be  used  to  handle  the  U.S.  Air  Force  Europe  and  Pacific  Air  Force 
bases  if  their  RPC  is  out  of  commission. 

Each  of  the  systems  have  or  have  proposed  a  disaster  recovery  plan  that  would 
provide  continuity  of  service.  Evaluation  of  the  plans  from  three  perspectives  follows: 

•  Extent  of  support  None  of  the  three  plans  permit  the  maintenance  information 
management  system  to  operate  without  some  degradation  of  service.  The 
nCARRS  proposal  would  require  the  bases  to  use  the  system  on  a  scheduled 
basis,  for  example,  according  to  geographic  location  to  spread  out  the 
workload,  and  limit  or  eliminate  standard  reporting  for  the  duration  of  the 
emergency.  The  RPC/CAMS  plan  would  degrade  each  of  the  remaining  RPCs 
such  that  service  would  be  reduced  and/or  deferred  during  the  emergency.  The 
REMIS  plan  would  also  limit  service  to  accepting  maintenance  data,  and 
severely  restricting  or  eliminating  the  generation  of  standard  and  REMISTALK 
reports. 

•  Effectiveness  of  support.  A  key  ingredient  in  providing  support  is  the 
availability  of  technical  personnel  to  operate  and  manage  the  system.  If  a 
disaster  struck  the  operational  centers  of  these  systems,  the  people  operating 
and  managing  the  systems  could  be  unable  to  continue  due  to  injury.  Both 
TICARRS  and  REMIS  are  centrally  managed  by  a  relatively  small  team  of 
operational  staff  and  would  be  hard-pressed  to  replace  them.  Other  similarly 
skilled  personnel  in  the  organizations  could  be  used,  but  they  would  not  likely 
be  familiar  with  the  operation  of  the  respective  systems.  Since  CAMS  would 
be  installed  in  other  RPCs,  which  were  also  operating  CAMS,  there  would  be 
little  or  no  skill  or  training  problem  in  supporting  the  bases  from  an  operational 
viewpoint. 

•  Execution  of  support.  Of  the  three  plans  only  the  CAMS  plan  at  the  RPCs  has 
been  tested.  TICARRS  has  not  had  the  need  or  opportunity  to  do  so,  and 
REMIS,  not  yet  fully  operational,  has  not  yet  taken  the  opportunity  to  conduct 
such  a  test 
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f.  Summary 

The  purpose  of  examining  the  system  and  technology  related  to  operating 
characteristics  is  to  understand  the  relative  strengths  and  weaknesses  of  each  system 
beyond  functional  capability  or  scope  of  service.  The  results  of  this  analysis  are  an 
important  factor  in  determining  the  effort  and  expenditures  that  must  be  applied  to  each 
system  to  provide  a  satisfactory  weapons  maintenance  information  system,  regardless  of 
which  system  is  ultimately  used. 

In  addition  to  the  issues  of  requiring  additional  functional  capability  and  supporting 
additional  weapons  and  equipment,  it  will  be  necessary  for  the  TICARRS  central  computer 
system  to  be  significantly  expanded.  If  TICARRS  is  to  undertake  the  transaction  workload 
currently  processed  by  CAMS,  it  will  need  to  be  about  four  times  more  powerful  than  the 
currently  installed  system  at  Andover,  Massachusetts.  This  increase  would  permit 
TICARRS  to  handle  all  the  CAMS  transactions  and  data  reporting  functions  provided  by 
REMIS.  The  system  availability  and  performance  exhibited  by  the  TICARRS  system  are 
adequate,  as  is  the  operational  management  control.  It  will  be  necessary  for  the  TICARRS 
system  to  maintain  the  same  level  of  performance  if  selected  to  support  all  the  weapons, 
bases,  and  users.  Finally,  a  back-up  disaster  recovery  process  would  need  to  be  installed  to 
ensure  continuity  of  service. 

Two  key  system  technology  and  performance  issues  affect  our  evaluation  of 
REMIS.  The  first  is  the  system  performance  of  REMIS.  While  the  simple  transaction 
response  time  is  adequate,  effort  must  be  expended  to  balance  the  central  computer 
workload  to  provide  better  turnaround  time  for  those  users  requesting  standard  and 
REMISTALK  reports.  In  addition,  the  programs  that  provide  operations  support  and  those 
that  process  input  from  CAMS  (SSI)  must  be  examined  to  see  if  the  amount  of  computer 
resources  used  can  be  reduced.  Ultimately,  additional  computing  power  will  be  necessary 
to  handle  the  full  workload  expected  by  REMIS.  The  second  issue,  related  to  the  first,  is 
that  the  standard  report  capability  of  REMIS  must  be  extended,  and  the  REMISTALK 
program  must  be  expanded  to  allow  a  broader  scope  of  data  to  be  included  in  its  reporting 
capability.  While  extending  the  scope  of  REMISTALK  is  not  a  technical  challenge  in  itself, 
doing  so  without  exposing  the  system  to  more  performance  problems  will  require 
significant  investment  in  analysis  and  design  at  the  system  level.  The  REMIS  disaster  back¬ 
up  plan  must  be  tested  to  verify  the  ability  of  REMIS  to  support  the  Air  Force  under 
disaster  conditions. 

CAMS  suffers  from  poor  availability  and  poor  transaction  response  time  at  many 
bases.  Unfortunately,  it  is  unclear  how  much  of  the  problem  is  tied  to  the  CAMS  software. 
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how  much  is  due  to  operational  difficulties,  and  how  much  to  communication  system 
problems.  The  evidence  indicates  that  CAMS  has  good  performance  and  good  availability 
at  a  few  bases,  which  leads  to  the  conclusion  that  without  top-notch  ADP  and  data  base 
manager  support,  CAMS  may  continue  to  see  the  same  level  of  performance  and 
availability.  The  move  to  RPCs  will  tend  to  improve  operational  control  and  discipline. 
CAMS  data  base  managers  will  need  to  remain  at  the  bases  as  planned,  and  the  institution 
of  a  CAMS  representative  will  be  helpful.  The  reliability  of  CAMS  software  needs  to  be 
examined  to  ensure  it  is  not  the  basis  of  CAMS  availability  or  performance  problems,  since 
the  RPC  experience  to  date  has  shown  that  the  RPC  computer  availability  is  not  a 
significant  problem.  It  is  possible  that  the  few  bases  with  good  performance  and  availability 
are  not  seeing  the  same  problems  as  the  majority  of  the  bases.  This  must  be  an  ongoing 
effort  of  the  CAMS  development  and  maintenance  staff. 

2.  Data  Integrity  and  Security 

A  maintenance  data  system  requires  procedures  to  ensure  that  only  authorized 
persons  can  enter  or  change  data.  According  to  the  Air  Force,  CAMS  meets  the 
requirements  for  a  Trusted  Security  Level  C2.  REMIS  is  also  required  to  meet  the  Level  C2 
requirements,  according  to  its  current  operational  requirements  document.  If  TICARRS 
were  to  become  the  standard  Air  Force  system,  it  would  presumably  be  required  to  meet  the 
same  requirements. 

The  SBLC  system  used  by  CAMS  is  vulnerable  to  tampering.  A  June  1992  Air 
Force  Audit  Agency  report  concluded  that  security  features  were  not  adequate  to  protect  the 
SBLC  communications  process,  the  SBLC  network,  the  SBLC  data  base,  and  remote 
terminals.  The  report  stated,  “under  carefully  controlled  conditions,  we  were  able  to  access 
the  SBLC  communications  processors  at  22  of  24  bases  tested  via  the  Etefense  Data 
Network  without  detection  and  without  using  an  SBLC  user  identification  code  or 
password.”  Inadequate  control  of  dial-up  access  could  lead  to  “overstating  the  quantity  of 
parts  actually  on  hand”  which  “could  affect  a  base’s  day-to-day  ability  to  meet  aircraft 
maintenance  schedules.”  Corrective  actions  are  under  way.  The  Air  Force  Audit  Agency 
stated:  “The  corrective  actions  taken  and  planned  are  responsive  to  the  issues  and 
recommendations  in  this  report”  Regionalization  will  mean  fewer  sites  to  secure,  but  will 
increase  the  importance  of  secure  communications  and  remote  terminals. 

REMIS  security  procedures  are  also  a  concern.  While  a  password  is  r^uired  in 
order  to  input  configuration  data,  once  a  person  gets  access  to  the  system,  he  or  she  can 
enter  configuration  data  on  any  system.  For  example,  a  KC-135  expert  could  alter  the 
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configuration  for  the  C- 141.  While  REMIS  developers  are  aware  of  this  difficulty,  they 
have  concluded  that  it  is  too  difficult  to  resolve  within  the  current  REMIS  architecture  and 
cost  constraints. 

REMIS  does  not  currently  meet  Level  C2  requirements,  because  its  Tandem 
platform  did  not  have  the  appropriate  security  product  available.  REMIS  is  currently  using 
a  software  called  ONGUARD  from  Computer  Associates.  Tandem  has  recently  released  a 
package  called  SAFEGUARD  that  meets  Level  C2  requirements.  Litton  is  working  to 
determine  the  cost  of  upgrading  to  SAFEGUARD,  but  the  required  resources  and  costs 
were  not  available  for  this  study. 

nCARRS  has  not  been  officially  certified  as  meeting  the  requirements  for  Trusted 
Security  Level  C2,  but  its  architecture  seems  easily  adaptable  to  meet  the  requirements.  The 
software  currently  has  the  process  controls  required  to  limit  access  and  capability  by 
authorized  users.  The  transaction  journals  provide  an  audit  trail  of  all  activity  against  the 
data  base  via  on-line  conversations.  Also,  whenever  a  user  creates  or  modifies  a  record,  the 
affected  record  contains  the  operator  identification  (ID)  plus  the  date  and  time  of  the  action. 
When  standard  queries  are  executed,  a  data  base  record  is  written  recording  query 
parameters  plus  the  operator  identification.  All  access  and  executions  initiated  via  the  Bull 
Hmesharing  Subsystem  are  also  traceable.  TICARRS  has  the  capability  to  set  the  time-out 
parameter  (time  after  which  the  system  disconnects  if  the  terminal  is  not  touched)  to  any 
level  required. 

Assigning  individual  operator  identifications  would  require  some  modification  to 
the  operator  identification-generation  program.  When  new  units  are  activated  under 
TICARRS  today,  the  group  operator  IDs  are  automatically  generated  from  the  organization 
structure.  That  program  would  require  modificaUon  to  generate  an  operator  ID  for  each 
person  loaded  to  TICARRS  under  the  Personnel  Management  Conversation.  Individuals 
would  be  given  a  default  set  of  permissions  based  on  their  organization  and  duty  Air  Force 
Specialty  Code.  Current  TICARRS  software  already  provides  for  the  delivery  and 
maintenance  of  the  operator  IDs.  The  operator  ID  data  field  provides  for  the  ger.eration  of 
over  60  million  passwords  using  just  alpha  and  numeric  characters,  so  no  changes  are 
required  in  the  data  field  size.  The  number  of  personnel  accessing  TICARRS  from 
locations  other  than  units  is  usually  small,  and  many  of  them  already  have  an  individual 
operator  ID  assigned.  Approximately  100  labor  hours  would  be  required  to  modify  the 
existing  operator  ID-generation  program. 
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3.  Ease  of  Use 


Many  characteristics  play  a  role  in  determining  the  ease  of  use  of  an  information 
system.  The  following  are  among  them: 

•  difficulty  in  learning  to  effectively  use  the  system, 

•  adequacy  of  training,  including  manuals  and  documentation, 

•  presence  and  type  of  help  facilities, 

•  presence  of  error  checking,  completeness  of  checking,  and  usefulness  of  error 
messages, 

•  need  for  multiple  data  entry, 

•  ease  of  obtaining  reports,  both  standard  and  ad  hoc, 

•  consistency  of  the  user  interface, 

•  ability  to  input  and  output  data  easily  and  quickly,  and 

•  ability  to  modify  data. 

We  used  three  primary  sources  of  information  to  address  the  issue  of  ease  of  use 
for  CAMS/REMIS  and  TICARRS: 

(1)  IDA  team  members’  observations  and  discussions  with  users, 

(2)  responses  to  a  survey  during  the  Operational  Assessment  of  TICARRS  92,  and 

(3)  answers  to  structured  questions  posed  to  the  REMIS  developers  and  to  the 
F-15  SPO  (which  uses  TICARRS). 

a.  Observations  of  and  Discussions  With  Users 

Throughout  the  course  of  this  study  effort,  the  IDA  team  made  deliberate  and 
extensive  observations  on  the  operations  of  CAMS,  REMIS,  and  TICARRS  at  base,  depot, 
MAJCOM,  and  SPO  levels. 

Base-level  observations  of  users  (Langley,  HUl,  Holloman,  and  Seymour  Johnson) 
provided  mixed  opinions  about  the  two  information  systems — ^not  a  surprise  since  bases 
have  some  latitude  in  organizing  and  implementing  maintenance  policies  and  procedures. 
Our  observations  at  the  bases  provided  the  following  information: 

•  CAMS  is  considered  slow  and  difficult  to  use,  requiring  more  screens  than 
TICARRS  for  many  functions, 

•  CAMS  served  the  ctebrief  function  adequately, 

•  engiiK:  management  was  completely  satisfied  with  CAMS  at  Langley  AFB, 

•  training  and  motivation  for  CAMS  is  insufficient. 
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•  CAMS  was  sufficient  for  scheduling  and  management, 

•  REMIST ALK  is  slow  in  retrieving  information,  and  some  users  were  not 
satisfied  with  the  level  of  training, 

•  nCARRS  was  easier  to  use  than  CAMS, 

•  TICARRS  requires  DRC  support  for  some  minor  changes  to  the  system, 

•  some  backshop  personnel  thought  TICARRS  required  too  many  screens  to 
retrieve  information,  relative  to  CAMS,  and 

•  depot  maintenance  personnel  found  that  TICARRS  conveniently  provided  the 
necessary  information  to  support  the  two-level  maintenance  test  at  Ogden. 

Our  observations  at  one  MAJCOM  (Air  Combat  Command)  yielded  the  following 
information: 

interface  problems  between  CAMS  and  REMIS  are  serious  in  the  areas  of 
status,  inventory,  and  utilization, 

•  REMIS  has  problems  downloading  data, 

•  REMIS  is  slow, 

•  REMIS  cannot  easily  provide  raw  data  for  analysis,  and 

•  REMIS  adequately  supports  the  logistics  community. 

No  other  MAJCOMS  were  solicited  for  information  on  the  scale  of  ACC. 

ALCs  seem  to  have  success  using  REMIS  and,  since  few  of  their  requirements  are 
extremely  time-sensitive,  some  delays  in  getting  information  are  not  critical.  In  general 
persoimel  at  the  ALCs  had  the  following  to  say: 

•  REMIS  is  user-friendly  and  improving  over  time, 

•  REMIS  handles  communications-electronics  and  space  equipment  adequately, 

•  REMIS  might  be  losing  40%-50%  of  the  data  it  should  have,  and 

•  training  on  REMIS  is  poor. 

The  SPOs  have  an  easier  time  accessing  and  using  TICARRS  than  REMIS. 
Contractors  have  the  following  opinions: 

•  REMIS  suffers  from  poor  access,  poor  training,  and  inability  to  retrieve  data 
easily  and  quickly, 

•  They  prefer  working  with  raw  t^)es  of  product  performance  data, 

•  nCARRS’s  retrieval  of  narratives  and  other  select  information  is  relatively 
easy, 

•  They  get  better  support  for  TICARRS  than  for  REMIS. 
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b.  Formal  Survey  Taken  at  Operational  Assessment  of  TICARRS  92 

During  the  Operational  Assessment  of  TICARRS  92,  a  number  of  questions  were 
asked  of  the  Seymour  Johnson  personnel  about  the  ease  of  use  of  CAMS,  of  TICARRS, 
and  of  TICARRS  compared  to  CAMS.  Responses  were  categorized  by  generic  work  center 
(flight  line,  backshop,  stafO  and  paygrade  (E-1  to  E-3,  E-4  to  E-5,  E-6  and  above).  The 
specific  measurement  criteria  were  ease  of  use,  ability  to  input  easily,  ability  to  output 
easily,  ability  to  modify  data,  usefulness  of  training,  usefulness  of  manuals  and 
documentation,  ability  to  input  quickly,  and  ability  to  output  quickly. 

Tables  V-6  and  V-7  show  the  percentage  of  respondents  who  were  either  satisfied 
or  very  satisfied  with  an  aspect  of  system  operation  (aggregated  together  as  satisfied)  as  a 
percentage  of  those  who  had  an  opinion  (including  those  who  responded  mixed/neither). 
The  tables  also  show  the  percentage  who  were  either  dissatisfied  or  very  dissatisfied 
(aggregated  together  as  dissatisfied). 

Table  V-6  shows  the  responses  for  the  pre-assessme. .  survey  of  CAMS  users. 
Between  23  and  ^  '  nercent  of  the  of  the  flight-line  respondents  were  satisfied  with  these 
features,  while  between  33  and  50  percent  were  dissatisfied.  In  the  backshops,  satisfaction 
ranged  from  32  to  46  percent  and  dissatisfaction  from  23  to  34  percent.  Staff  satisfaction 
ranged  from  29  to  60  percent  and  dissatisfaction  from  18  to  40  percent.  For  all  paygrades 
and  woric  centers  combined,  those  satisfied  or  very  satisfied  ranged  from  32  to  45  percent; 
those  dissatisfied  or  very  dissatisfied  ranged  from  26  to  41  percent 

A  large  number  of  respondents  (30  percent  for  overall  ease  of  use)  gave  responses 
that  indicated  indifference  to  the  ease  of  use  of  the  information  system. 

Although  paygrade  distinctions  are  not  shown  in  the  table,  lower  pay  grade 
personnel  had  satisfaction  levels  between  35  and  44  percent.  E-4s  and  E-5s  ranged  from  30 
to  48  percent  E-6  and  above  ranged  from  27  to  44  percent 

While  there  were  variations  by  work  area  and  pay  grade,  the  ability  to  generate 
output  was  the  most  liked  feature  of  CAMS.  Speed  of  input  and  the  ability  to  change  data 
were  the  most  disliked. 

Table  V-7  displays  users’  opinions  about  the  ease  of  use  of  TICARRS,  derived 
from  responses  to  a  survey  taken  during  the  sixth  week  of  the  Operational  Assessment 
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Table  V*6.  Results  of  Pre-Assessment  Survey  of  CAMS  Users 


Criteria 

Response 

Flight  line 

Backshop 

Staff 

All 

Ease  Of  use 

stuisfted 

37% 

42% 

60% 

42% 

dissatisfied 

35% 

24% 

23% 

28% 

Ability  to  Input  Easily 

satisfied 

32% 

44% 

60% 

41% 

dissatisfied 

40% 

25% 

23% 

31% 

AinliQr  to  Output  Easily 

satisfied 

42% 

46% 

52% 

45%' ' 

dissatisfied 

33% 

23% 

18% 

26% 

AbiiiQF  to  Modify  Data 

satisfied 

23% 

35% 

42%  _ 

30%,, 

dissatisfied 

50% 

33% 

40% 

41% 

Usefulness  of  Trainnig 

s^isfied 

27% 

38% 

47% 

34%7' 

dissatisfied 

34% 

28% 

28% 

31%' 

UselEoiness  of  Manuals  and 

satisfied 

29% 

32% 

42% 

32%  ''' 

Docmnentatioa 

dissatisfied 

40% 

30% 

30% 

34% 

Ataiity  io  Input  Quickly 

satisfied 

27% 

40% 

43% 

,35%.;;' 

dissatisfied 

50% 

33% 

30% 

40% 

Ability  to  Ougiut  Quickly 

satisfied 

30% 

35%  , 

29% 

32%'"' 

dissatisfied 

45% 

34% 

37% 

39% 

Table  V-7.  Opinions  Concerning  TICARRS  In  Six- Week  Survey 


Criteria 


Easaofase 

Abi%  to  Xi^t  Hazily 
to  Ovitpnt  Easily 


lisesfblnesaofTraittiag 

Docaraentatkai 


Response  Flight  line  Backshop 


Staff 


All 


sausTied 

dissatisfied 

satisfied 

dissatisfied 

satisfied 

dissatisfied 


41% 


26%  28% 

'  38%.,  ..  4I%,„: 


25%  29% 

37%  37% 


<  .> 


x-X-X*: 

fM 


dissatisfied 

dissatisfied 


dissatisfied 


dissatisfied 


dissatisfied 


,47%  ,,37%^, 

21%  33% 

47%  r'' 

25%  33% 

31%  40% 

32%  40%  33%  36% 

29%  27%  22%  ,.,2?^ 

29%  40%  38%  36% 


27%  33%  26%  30% 

'  45%  ^34%'-.  ':  31%-Jl:.« 

29%  33%  33%  31% 

. 

41% 


42%  ,  ,>v25% 21% 


32% 


39% 


37% 
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A  comparison  of  Tables  V-6  and  V-7  reveals  that: 

•  Flight-line  personnel  were  generally  more  satisfied  with  TICARRS  in  the  ease- 
of-use  categories  while  the  backshops  were  consistently  less  satisfied  with 
TICARRS.  Staff  was  decidedly  less  satisfied  with  TICARRS. 

•  CAMS  was  preferred  for  ability  to  output  easily  and  usefulness  of  training, 
while  TICARRS  was  preferred  for  ability  to  input  quickly.  Most  other 
measures  showed  mixed  results. 

•  Again,  there  was  a  large  percentage  of  respondents  who  are  indifferent  about 
these  systems. 

•  Overall  satisfaction  with  the  two  systems  regarding  ease  of  use  seems 
remarkably  similar.  Of  the  respondents,  42  percent  were  satisfied  with  CAMS, 
and  28  percent  were  dissatisfied;  41  percent  were  satisfied  with  TICARRS, 
and  28  percent  dissatisfied. 

Comparison  of  survey  results  taken  six  weeks  apart  may  not  tell  the  whole  story. 
People’s  impressions  of  CAMS  may  have  been  changed  by  their  use  of  TICARRS.  In 
order  to  get  a  clearer  picture  of  attitudes  at  the  end  of  the  Operational  Assessment, 
respondents  were  asked  to  directly  compare  TICARRS  and  CAMS.  People  were  asked 
which  system  they  preferred  regarding  the  same  eight  ease-of-use  categories.  Their 
responses  are  tabulated  in  Table  V-8. 

Table  V-8  shows  a  mild  preference  for  CAMS.  There  is  variation  by  work  center. 
Flight-line  personnel  preferred  TICARRS  for  overall  ease  of  use  by  40  percent  to  30 
percent  though  higher  percentages  preferred  some  specific  aspects  of  CAMS  than 
TICARRS.  High  paygrade  flight-line  personnel  (not  shown  in  the  table)  preferred 
TICARRS  for  overall  ease  of  use  by  better  than  two  to  one.  Backshop  and  staff  personnel 
showed  a  general  prefeience  for  CAMS. 

Aggregating  over  all  work  centers  and  paygrades,  CAMS  was  preferred  by  39 
percent  "'hile  TICARRS  was  preferred  by  31  percent.  Thirty  percent  had  no  preference. 

One  of  the  consistent  trends  throughout  the  surveys  is  the  large  number  of 
respondents  who  had  no  preference  for  either  system.  These  values  ranged  from  about  20 
percent  to  over  50  percent  in  some  cases. 

It  is  worth  noting  that  TICARRS  was  favored  by  more  people  during  the  fourth 
week  of  the  Operational  Assessment  than  during  the  sixth  week.  This  is  demonstrated  in 
Table  V-9.  During  the  fourth  week,  TICARRS  was  preferred  to  CAMS  among  survey 
respondents  for  ease  of  use  by  41  percent  to  28  percent.  There  was  a  particularly  marked 
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difference  in  the  responses  of  backshop  personnel,  who  favored  TICARRS  40  percent  to 
31  for  ease  of  use  in  week  four.  During  week  six  they  favored  CAMS  by  44  percent  to  26. 


Table  V-8.  Comparison  of  CAMS  and  TICARRS  in  Sixth-Week  Survey 


Criteria 

Response 

Flight  line 

Badcshop 

Staff 

All 

Base  of  use 

CAMS  preferred 

30% 

44% 

39% 

39% 

TICARRS  preferred 

40% 

26% 

23% 

31% 

AbSior  to  lupot  Easily 

CAMS  preferred 

34% 

39% 

19% 

36% 

TICARRS  preferred 

38% 

27% 

33% 

32% 

AbOhy  to  Output  Easily 

CAMS  preferred 

38% 

47% 

47% 

44% 

TICARRS  preferred 

34% 

20% 

19% 

25% 

AbtfHy  to  Modify  Data 

CAMS  preferred 

37% 

45% 

30% 

40% 

TICARRS  preferred 

34% 

20% 

34% 

26% 

th^tness  of  Training 

CAMS  preferred 

29% 

38% 

37% 

35% 

TICARRS  preferred 

24% 

22% 

17% 

22% 

UitoMncSSof  Mamials  and 

CAMS  preferred 

26% 

32% 

33% 

30% 

Pocuntoaiation 

TICARRS  preferred 

30% 

28% 

16% 

27% 

Ability  to  htput  Qaickiy 

CAMS  prefated 

32% 

38% 

36% 

36% 

Ability  to  On^  Quickly 

TICARRS  preferred 

37% 

30% 

28% 

32% 

CAMS  preferred 

34% 

42% 

40% 

39% 

TICARRS  preferred 

33% 

23% 

17% 

26% 

Table  V-9.  Comparison  of  CAMS  and  TICARRS  In  Fourth-Week  Survey 


Criteria 

Response 

Flight  line 

Backshop 

Staff 

All 

Eascofuso;' 

CAMSpn^aned 

24% 

31% 

34%  , 

TICARRS  preferred 

47% 

40% 

24% 

41% 

Abilt^  to  Input  Easily 

CAMS  preferred 

23% 

27% 

38%  , 

26% 

TICARRS  preferred 

48% 

40% 

28% 

43% 

Abibiy  to  Output  Easily 

CAMS  preferred 

26% 

37% 

39% 

32% 

TICARRS  preferred 

42% 

28% 

19% 

34% 

AbiEiy  to  Modify  Doa 

CAMS  preferred 

30% 

35% 

41% 

33%  „ 

TICARRS  preferred 

41% 

29% 

31% 

35% 

IlseriilttessofTtadtting 

X&efbbKSS  of  Manuals  and 
VocmeotaHoa 


CAMSpte&ned  m 

TICARRS  preferred  35% 

CAMSprefeiftid  20% 

TICARRS  preferred  41% 

•  CAMSpnfisred  '  25% 

TICARRS  preferred  44% 

CAMSjitefeneil  '  26% 
TICARRS  tJreferr^  40% 


h  16%  30% 

h} :  ^ 

b  29%  37% 

(,  27%  39% 


29%  23%  34% 


I 

I 

I 
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At  least  five  factors  may  have  influenced  the  apparent  shift  away  from  TICARRS 
between  week  four  and  week  six: 

•  Many  of  DRC’s  support  personnel  were  removed  from  the  base  after  the  fourth 
week.  Without  the  extra  help,  users’  opinions  of  TICARRS  may  have 
suffered.  (On  the  other  hand,  no  overwhelming  demand  for  the  service  of  the 
remaining  DRC  reprCv'entatives  was  observed  during  the  last  two  weeks.) 

•  The  4th  Wing  was  undergoing  a  surge  exercise  during  the  sixth  week.  The 
extra  burden  of  using  a  still  relatively  unfamiliar  system  in  a  stressful 
environment  may  have  weighed  against  TICARRS. 

•  Wing  personnel  may  have  been  less  serious  about  the  Operational  Assessment 
during  the  sixth  week.  They  knew  they  were  going  to  get  CAMS  back  very 
soon.  Perhaps  the  desire  to  get  the  whole  thing  over  with  colored  their  attitudes 
toward  TICARRS. 

•  The  fourth-week  survey  had  a  better  response  rate  than  the  sixth-week  survey: 
1,055  people  responded  to  the  former  (out  of  a  relevant  wing  population  of 
1,322)  and  751  responded  to  the  latter.  The  fourth-week  survey  results  may  be 
more  representative  of  the  entire  wing’s  opinions. 

•  Upon  further  reflection,  some  users  may  have  decided  that  CAMS  was  easier 
to  use. 

On  balance,  it  is  probably  best  to  view  the  Operational  Assessment  of  TICARRS  92 
as  showing  no  marked  preference  for  either  system  with  respect  to  ease  of  use.  (Only  a 
slight  preference  for  CAMS  was  shown  at  the  end  of  the  assessment.)  The  fourth-week 
survey  favored  TICARRS  and  the  sixth-week  survey,  CAMS.  Some  groups  of  users 
found  TICARRS  more  user-friendly,  others  preferred  CAMS,  many  were  indifferent. 

It  should  be  remembered  that  some  of  the  apparent  strengths  of  TICARRS  could 
not  be  well-appreciated  during  the  Operational  Assessment.  Its  ability  to  keep  track  of 
aircraft  configuration  information  was  compromised  by  the  inability  to  load  comprehensive 
historical  data  from  CAMS.  Some  of  this  was  CAMS’s  fault  and  some  was  not.  The 
incompleteness  of  the  historical  data  also  may  have  made  TICARRS  appear  less  useful  (and 
less  user-friendly)  to  users  of  the  data  at  the  wing  level  than  it  would  have  looked  after 
more  extensive  operation  (which  would  have  populated  the  data  base). 

Of  course,  many  of  the  ultimate  users  of  maintenance  information  do  not  reside  at 
the  wing.  We  arc  really  interested  in  comparing  the  user-fnendliness  of  TICaRRS  with  that 
of  CAMS/REMIS.  The  Operational  Assessment  only  addressed  the  comparison  between 
TICARRS  and  CAMS. 
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c.  Response  to  Structured  Questions 

IDA  requested  the  same  information  about  the  number  of  maintenance  actions, 
mean  time  between  maintenance  actions,  and  other  product  performance  data  by  aircraft 
type,  by  work  unit  code  (WUC),  and  by  part  number  from  both  the  REMIS  developer  and 
from  a  TICARRS  user  (the  F-15  SPO).  The  idea  was  to  compare  the  ease  of  use  and 
functionality  of  the  two  systems.  TICARRS  was  able  to  provide  all  the  requested 
information  in  a  straightforward  way.  On  the  other  hand,  the  REMIS  experts  were  often 
unable  to  provide  the  desired  information,  delivering  information  on  different  aircraft  or  for 
different  time  periods  than  the  ones  addressed  in  the  request 

In  addition,  standard  REMIS  queries  were  sometimes  unable  to  provide  summary 
information  without  a  great  deal  of  unrequested  detail.  The  only  way  to  avoid  the  detail  was 
to  use  the  more  time-consuming  mechanism  of  REMIST ALK. 

d.  Conclusions 

Despite  some  strong  complaints  about  the  ease  of  use  of  CAMS  by  wing-level 
personnel,  the  Operational  Assessment  of  TICARRS  92  did  not  show  TICARRS  to  be 
consistently  easier  to  use  for  such  users.  Ease  of  use  is  probably  best  considered  roughly 
equal  at  the  wing  level. 

Overall,  when  we  looked  beyond  the  wing  level,  we  found  greater  satisfaction  with 
the  ease  of  use  of  TICARRS  than  with  REMIS.  While  there  were  instances  of  satisfaction 
with  the  ease  of  use  of  REMIS  at  the  depot  level,  there  were  other  instances  of 
dissatisfaction.  MAJCOM  personnel,  weapon  system  SPOs,  and  contractor  personnel 
indicated  preferences  for  TICARRS. 

D.  DATA  ACCURACY  AND  COMPLETENESS 

This  section  discusses  the  accuracy  of  Air  Force  maintenance  data,  available  in 
CAMS/REMIS  and  TICARRS.  As  currently  operated,  the  systems  are  interdependent  in 
many  respects.  For  example,  any  inaccurate  CAMS  data  will  be  fed  to  both  REMIS  and 
TICARRS  (as  well  as  other  systems),  leading  to  complaints  of  inaccurate  data  from  these 
systems.  For  TICARRS,  there  exists  some  information  concerning  data  accuracy  of  the 
TICARRS  direct-entry  system  (which  would  become  standard  if  TICARRS  became  the 
basic  Air  Force  information  system). 

It  is  not  possible  to  compare  the  output  of  the  two  systems  with  an  independent 
authoritative  source  of  accurate  data — the  systems  are  designed,  in  most  cases,  to  be  the 
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authoritative  source.  However,  we  examined  evidence  on  the  performance  of  each  of  the 
systems  relative  to  data  accuracy  and  validity.  We  included  information  we  gathered  and 
information  provided  to  us  by  users  and  developers  of  the  systems. 

It  is  easier  to  gather  evidence  on  data  accuracy  and  validity  for  CAMS/REMIS, 
since  it  operates  as  the  Air  Force’s  standard  maintenance  data  system,  than  it  is  for 
TICARRS.  The  data  are  being  loaded  and  processed  in  CAMS/REMIS,  while  TICARRS 
currently  accepts  data  through  CAMS  for  two  weapon  systems,  the  F-i5  and  F-16.  One 
limited  body  of  evidence  for  TICARRS  comes  from  the  Operational  Assessment  at 
Seymour  Johnson  AFB.  Other  information  pertaining  to  the  accuracy  of  TICARRS  data 
dates  from  the  period  when  TICARRS  was  being  fed  directly  by  users. 

Information  on  the  accuracy  and  validity  of  data  in  the  maintenance  data  systems 
was  derived  from  multiple  sources.  While  there  is  significant  anecdotal  evidence  to  support 
problems  in  data  accuracy  with  these  systems,  we  have  tried  to  limit  our  assessment  to 
cases  where:  (1)  the  problem  is  clearly  visible  and  significant  and  (2)  there  is  empirical 
information  to  support  our  assessment.  In  the  subsections  that  follow  we  first  discuss 
CAMS/REMIS,  then  we  cover  TICARRS,  providing  information  comparing  data  accuracy 
under  TICARRS  direct-entry  to  CAMS.  We  next  address  data  integrity  and  security  under 
all  three  systems.  Finally,  we  summarize  our  assessment  of  the  systems. 

1.  CAMS/REMIS 

As  was  noted  in  the  discussion  of  data  base  type,  range,  and  organization,  data 
accuracy  and  validity  are  affected  by  the  CAMS/REMIS  system  architectures  and  data  base 
designs  and  interfaces.  Gaps  are  created  in  the  maintenance  data  that  must  be  filled  by 
personnel  re-entering  the  data.  Experience  indicates  that  they  often  do  not  re-enter  it 

It  should  be  noted  that  some  of  the  REMIS-reject  problem  is  not  intrinsic  to  the 
system  architecture,  but  rather  follows  from  the  Air  Force’s  decision  to  relax  the  stringency 
with  which  CAMS  edits  entries  without  relaxing  the  stringency  of  REMIS  edits.  Simple 
data-entry  errors  are  not  picked  up  by  CAMS,  but  are  flagged  by  REMIS.  This  generates 
the  same  kinds  of  gaps  in  the  data  base  already  described. 

a.  RECFU  n  and  Its  Effect  on  Data  Accuracy 

Altering  software  to  fix  problems  or  provide  new  capabilities  can  cause  new  errors 
to  arise  in  other  areas.  A  recent  example  is  the  RECFU  H  release  designed  to  allow  CAMS 
to  send  hourly  updates  to  the  EIMSURS  portion  of  REMIS  via  DDK  rather  than  daily 
updates  through  Autodin.  The  release  had  defects  that  involved  incorrect  editing  and 
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resulted  in  data  rejects  by  EIMSURS/REMIS.  The  data  rejects  then  caused  more  rejects  of 
records  that  were  based  on  the  ones  rejected,  leading  to  missing  data.  For  a  number  of 
reasons,  these  data  were  not  corrected  by  CAMS  users;  therefore,  the  status,  inventory, 
and  utilization  records  had  gaps. 

EIMSURS  was  also  rounding  flying  hours  differently  from  CAMS.  This 
aggravated  another  problem,  the  issue  of  gain/loss  data  when  aircraft  transfer  or  when  parts 
are  in/out  of  the  depot.  The  receiving  group  would  record  a  different  time  of  ownership 
than  the  sending  group,  creating  an  error  condition  since  the  times  did  not  match.  Of 
course,  the  error  condition  caused  any  subsequent  records  to  be  rejected  until  the  time  issue 
was  resolved,  resulting  in  more  data  gaps.  ACC  personnel  reported  inaccurate  data  being 
drawn  from  EIMSURS.  Litton  reported  that  the  problem  has  been  corrected. 

Finally,  there  is  an  issue  of  incorrect  data  in  the  interface  between  CAMS  and 
REMIS  which  caused  more  rejects  by  REMIS.  Several  steps  have  been  taken  to  remedy 
this  situation.  First,  the  defects  in  RECFU  n  are  being  fixed  and  tested.  Second,  some  of 
the  edits  have  been  changed  so  that  the  record  is  not  rejected,  but  the  data  items  with 
inconsistencies  are  flagged  to  warn  future  users.  In  addition,  the  MAJCOMS  have  been 
given  the  ability  to  review  and  correct  status,  inventory,  and  utilization  records  if  the 
reporting  bases  do  not  do  so.  Finally,  the  REMIS  PMO  is  directing  a  manual  loading  of 
base-level  CAMS  data  to  force  CAMS  and  REMIS  to  match.  The  REMIS  data  back  to 
November  1992  are  being  wiped  out,  and  CAMS  data  are  being  entered  manually.  The 
PMO  reported  that  this  activity  was  completed  on  schedule  in  June  1993, 

Corrective  actions  for  repair  of  RECFU  n  address  some  of  the  problems  with  the 
interface  between  CAMS  and  REMIS,  but  they  do  not  address  the  basic  problem  of 
incorrect  input  from  the  CAMS  user.  Instead,  REMIS  will  have  to  rely  on  the  CAMS  user 
(or,  more  likely,  the  data  base  manager)  or  the  technician  at  the  Air  Logistics  Center  to 
correct  errors  after  they  show  up  in  REMIS.  If  this  process  does  not  work,  the  Major 
Commands  can  change  the  data  in  REMIS,  but  these  changes  are  not  passed  back  to 
CAMS.  The  Major  Commands  have  agreed  to  use  the  manual  correction  capability  in  a 
minimal  way,  according  to  the  REMIS  PMO.  It  is  likely  that  the  problem  will  eventually 
recur,  unless  the  CAMS  users  go  back  and  fix  their  data,  which  they  historicaUy  have  not 
done.  It  is  our  assessment  that  the  regionalization  of  the  computer  systems  may  make  this 
worse,  since  it  involves  a  large  reduction  in  the  number  of  data  base  managem,  and  the 
remaining  data  base  managers  may  have  less  time  to  correct  CAMS  «Tors. 
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b.  Data  Accuracy  in  CAMS — Engines 

It  is  possible  lo  achieve  accuracy  in  CAMS  with  sufficient  time,  effort,  and 
management  attention,  as  demonstrated  by  the  following  example.  Engine  components  are 
critical  for  safety  of  flight,  and  APR  400-1  and  ACC  66-5  require  a  formal  reconciliation  of 
CAMS  and  CEMS  engine  data  every  six  months  for  every  base.  First,  the  base  data  base 
manager  does  a  delete  history  run  so  that  old  parts  removed  from  the  engine  and  shipped 
elsewhere  do  not  show  up  as  uninstalled.  This  cleans  the  data  base.  The  data  base  manager 
then  runs  a  backup  or  mirror  image  of  the  data  base  at  a  specified  time.  The  CAMS  tape  is 
sent  to  Tinker  AFB  by  express  courier.  Tinker  personnel  conduct  the  comparison  and  send 
the  base  a  report  of  the  comparison,  indicating  where  in  CAMS  corrections  need  to  be 
made.  For  major  modules  of  the  engine  (1534  reportable)  and  serially-tracked  items  (D042 
reportable)  that  are  reported  in  CAMS  and  sent  to  CEMS,  the  error  rate  is  generally  less 
than  0.5  percent. 

c.  Coronet  Deuce  Experience 

According  to  information  collected  during  a  portion  of  Coronet  Deuce,  the  Air 
Forces’  two-level  maintenance  test,  there  are  times  when  not  all  maintenance  activities  have 
been  processed  through  CAMS  into  the  central  data  bases  of  REMIS  and  TICARRS.  As 
indicated  in  Table  V-10,  one  Coronet  Deuce  analyst  found  that,  during  a  two-week  period, 
49  of  the  170  LRUs  (29  percent)  shipped  to  the  depot  were  not  recorded  as  unrepairable  at 
the  base  level  in  the  central  data  base.  The  source  of  the  data  was  TICARRS,  which 
obtained  its  base-level  data  from  CAMS. 

Table  V-10.  Missing  CAMS  information  on 
Not-Repairabie-This-Station  Actions  During  Coronet  Deuce 


Base 

LRUs  Shipped 

LRUs  with 

No  NRTS*  Action 
in  Data  Base 

Hill 

14 

5 

Shaw 

45 

12 

Richmond 

9 

1 

Ramstein 

29 

2 

Eielson 

6 

2 

Buckley 

7 

6 

Sioux  Falls 

1 

0 

Moody 

42 

11 

Osan 

2 

1 

Eglin 

15 

9 

Total 

170 

49 

*  NRTS  means  not  repairable  this  station. 
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d.  Data  Loss  Problems 


Lockheed  Corporation  in  Fort  Worth,  Texas,  has  noted  major  losses  of  F-16 
maintenance  data  during  passage  of  data  from  CAMS  into  D056  and  into  REMIS.  Data 
were  not  flowing  from  CAMS  at  the  base  level.  As  presented  in  Table  V-1 1,  at  the  peak  of 
the  problem,  ten  bases  were  without  data  flow.  The  problem  affected  data  from  29  bases, 
one  of  them  three  times,  and  four  of  them  twice.  In  one  case,  data  flow  did  not  occur  for  13 
months.  In  addition,  duplicate  data  were  sometimes  transmitted  from  CAMS  to  REMIS. 

In  addition  to  analyzing  regular  maintenance  data,  Lockheed  Fort  Worth  conducted 
separate  analyses  of  F-16  TCTO  data  flow  from  CAMS  to  REMIS,  CAMS  to  TICARRS, 
and  from  CAMS  and  D056  to  D057G.  D057G  received  only  4.6  percent  of  the  records, 
and  REMIS.  only  33.6  percent.  TICARRS  received  98.2  percent  of  the  records.  CAMS 
personnel  said  that  much  of  the  problem  was  due  to  disruption  from  the  release  of  RECFU 
n  and  that  the  data  eventually  were  transmitted  to  D056  and  REMIS.  However,  Lockheed 
was  still  unable  to  retrieve  the  data  in  June  1993. 

e.  Experience  With  REMIS 

(1)  EIMSURS  Data.  According  to  the  results  of  our  test,  REMIS  seems  to 
provide  accurate  current  data  on  mission-capable  rates  and  possessed  hours.  Data  from  the 
4th  Wing  showed  very  close  correspondence  to  TICARRS  and  REMIS  data  in  this  area. 

(2)  Product  Performance  Subsystem  Data.  Litton  Computer  Services,  the 
REMIS  development  contractor,  reports  a  reject  rate  of  just  under  5  percent  for  on- 
equipment  aircraft  maintenance  data  for  the  period  October  31,  1992,  through  January  2, 
1993.  Reject  rates  have  increased  considerably  since  RECFU  II  (see  Table  V-12).  Aircraft 
systems,  arguably  the  most  important  data  point,  have  reject  rates  in  the  range  of  8  to  11 
percent.  Reject  rates  for  communication  electronics  and  support  equipment  are  expected  to 
improve  significantly  soon  due  to  recent  fixes.  Utton  has  set  a  goal  of  reducing  the  overall 
reject  rate  in  the  Product  Performance  Subsystem  (PPS)  to  7  percent  and  the  aircraft  reject 
rate  to  5  percent  or  less  by  November  1993. 

As  of  late  May  1993,  46  system-to-system  input  (SSI)  interfaces  were  operating 
without  problems.  However,  one,  CAMS  on-equipment  maintenance,  was  not  operating  at 
all,  so  that  none  of  that  data  was  reaching  PPS.  Hie  fix  for  that  problem  was  in  system  test 
Three  additional  SSIs  had  processing  logic  problems,  causing  loss  of  data. 
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Table  V-11. 

Missing  Data  Fiow  from  CAMS  to  D056 

and  REMIS 

Base 

Stop  Report  Date 

Fix  Report  Date 

Months  Missing 

Andrews 

2/22/93 

3/17/93 

0.83 

Bei^strom 

3/31/93 

5/6/93 

1.20 

Carswell 

4/14/93 

5/20/93 

1.20 

Charleston 

12/4/92 

1/28/93 

1.80 

Des  Moines 

3/9/93 

4/5/93 

0.87 

Edwacds 

3/3/93 

4/5/93 

1.07 

Edwards 

4/6/93 

4/28/93 

0.73 

Eglin  (Block  SO  data) 

2/19/93 

3/8/93 

0.63 

Eglin 

3/16/93 

4/14/93 

0.93 

Eielson 

2/22/93 

3/17/93 

0.83 

Ellington 

1/11/93 

3/17/93 

2.20 

Fort  Smith 

2/22/93 

3/17/93 

0.83 

Great  Falls 

4/26/93 

5/20/93 

0.80 

Hanoodc 

12/4/92 

1/28/93 

1.80 

Hill 

1/11/93 

4/5/93 

2.80 

Hill 

4/6/93 

4/28/93 

0.73 

Hill 

5/12/93 

6/12/93 

1.00 

Hulman 

1/11/93 

6/7/93 

4.87 

KeUy 

1/11/93 

3/17/93 

2.20 

Kirtland 

4/26/93 

6nm 

1.37 

Luke 

1/19/93 

3/8/93 

1.63 

Luke 

4/6/93 

4/28/93 

0.73 

McConnell 

11/18/92 

3/8/93 

3.67 

Mountain  Home 

3/9/93 

4/5/93 

0.87 

New  Orleans 

1/11/93 

2/19/93 

1.27 

New  Orleans 

3/9/93 

4/5/93 

0.87 

Niagara  Falls 

12/11/92 

1.57 

Peoria 

3/9/93 

4/5/93 

0.87 

Puerto  Rico 

12/1/92 

6/11/93 

6.33 

Richmond 

12/4/92 

1/28/93 

1.80 

Springfield 

2/15/92 

3/18/93 

13.10 

Tinker 

2/2/93 

3/8/93 

1.20 

Truax 

3/9/93 

4/5/93 

0.87 

Tucson 

1/11/93 

3/8/93 

1.90 

Wiight-Patteison 

2/16/93 

3/17/93 

1.03 

Source:  Lockheed  Corporation 
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Table  V-12.  Reject  Rates  in  PPS  for  Selected  Periods  in  1993 


Type  of  Equipment 

February  22- 
March  15 

March  17-AprU  15 

April 16-April  28 

Aircraft 

10.7% 

10.5% 

8.5% 

Cooununications-electronics 

36.9% 

47,9% 

58.9% 

Engines 

14.7% 

25.2% 

39.7% 

Support  equipment 

42.0% 

74.2% 

73.8% 

Precision  measurement  equipment 

59.4% 

NA 

NA 

Missiles 

5.6% 

7.0% 

45.5% 

Trainers 

87.5% 

97.7% 

93.5% 

Total 

20.7% 

18.7% 

15.7% 

Note:  NA  means  data  were  not  available. 


(3)  LRU  Removals  Data.  In  April  1993,  Lockheed  Corporation  compared 
LRU  removals  using  TICARRS,  REMIS  standard  query,  and  REMISTALK  for  a  sample 
of  20  different  WUCs,  ten  for  the  F-16A/B  and  ten  for  the  F-16C/D.  The  REMIS  queries 
were  performed  by  the  most  experienced  REMIS  user  at  Ogden  ALC.  Results  are  presented 
in  Figures  V-8  and  V-9. 
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Number  of  Removal  Actions  Per  Work  Unit  Code 

■  REMISTALX  0  TICARRS 
Note:  REMIS  standard  query  found  no  data. 


Figura  V-8.  Comparison  of  LRU  Ramoval  Actions  by 
Work  Unit  Coda  for  tha  F-16A/B 
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Number  of  Removal  Actions  Per  Work  Unit  Code 


■  REMISTALK  0  TICARRS 

Note:  REMIS  standard  query  found  no  data. 

Figure  V-9.  Comparison  of  LRU  Removal  Actions  by 
Work  Unit  Code  for  F-16C/D 


In  every  case,  no  data  were  found  in  the  REMIS  standard  query.  A  small  number  of 
maintenance  actions  (never  more  than  ten  for  any  given  work  unit  code)  were  found  in 
REMISTALK.  REMISTALK  response  times  were  in  excess  of  36  hours,  well  over  the 
required  standard  of  24  hours.  Only  the  TICARRS  data  seemed  to  provide  a  realistic 
number  of  removals  for  each  WUC. 

f .  Test  of  REMIS  Functionality  and  Selected  Operating 
Characteristics 

This  section  describes  the  results  of  an  IDA  test  of  REMIS  functionality  and  status 
performed  in  late  May  1993.  IDA  gave  the  CAMS/REMIS  PMO  and  the  F-15  SPO 
(TICARRS  users)  two  identical  data  requests  in  late  May  1993.  The  requests  covered 
several  aircraft  types,  and  the  data  requested  are  typical  of  those  we  had  seen  requested  by 
base,  ALC,  and  MAJCOM  users.  The  test  focused  on  both:  (1)  the  basic  functions  of 
REMIS  and  (2)  specific  areas  of  functionality  that  TICARRS  users  have  requested  be 
provided  before  TICARRS  would  be  fully  replaced  by  CAMS/REMIS. 

Both  REMIS  and  TICARRS  were  able  to  provide  basic  statistics — mission-capable 
rates  and  possessed  hours — for  the  F-15E  aircraft,  4th  Hghter  Wing,  at  Seymour  Johnson 
AFB.  Possessed  hours  agreed  within  1  percent  in  the  two  data  systems,  while  mission- 
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capable  rates  agreed  within  0. 1  percentage  point.  However.  REMIS  found  no  data  for 
March  and  April  of  1993.  Complete  results  of  the  test  are  included  in  Appendix  D. 

REMIS  did  not  provide  additional  Seymour  Johnson  data  we  asked  for  on  MTBFs 
and  MTBMAs. 

We  asked  for  data  on  mean  time  between  maintenance  actions  (MTBMA)  for  F-15s, 
and  both  REMIS  and  TICARRS  were  able  to  provide  reasonable  data.  The  TICARRS  data 
were  only  for  U.S.  aircraft,  not  F-15s  worldwide,  as  requested. 

We  asked  for  all  serial  numbers  within  WUC  74  (fire  control)  that  had  more  than 
three  maintenance  actions  during  the  first  quarter  of  1993.  TICARRS  was  able  to  provide 
this;  REMIS  provided  an  alternative  report  that  listed  the  ten  top  man-hour-consuming 
systems  for  the  F-15C  by  work  unit  code,  not  by  serial  number. 

We  also  asked  for  data  and  maintenance  narratives  on  the  F- 16  rotary  flap  actuator, 
a  part  that  is  tracked  by  TICARRS  for  warranty  purposes.  The  only  data  provided  by 
REMIS  were  maintenance  actions  and  man-hours  at  the  WUC  level.  TICARRS  provided 
68  complete  narratives  to  illustrate  its  capability. 

TICARRS  users  have  complained  that  the  algorithms  in  PPS,  EIMSURS,  and 
REMISTALK  for  calculation  of  mission-capable  (MC)  rates,  mean  time  between  repairs 
(MTBR),  and  mean  time  between  critical  failures  (MTBCF)  are  not  consistent  with  o..e 
another.  The  REMIS  PMO’s  explanation  was  that  different  groups  of  users  have  different 
methods  of  calculating  these  variables.  Developers  have  decided  to  wait  for  an  Air  Force 
policy  on  what  the  algorithms  should  be,  and  they  will  then  be  standardized  across  all 
modules. 

We  asked  for  several  items  frcm  GCSAS  in  the  second  request  GCSAS  was  able 
to  provide  a  list  of  current  TCTOs  for  the  B-1  (an  aircraft  not  supported  by  TICARRS),  but 
was  not  able  to  provide  a  list  of  approved  and  actual  configuration  for  the  same  aircraft 
because  the  actual  configuration  had  not  yet  been  loaded. 

We  asked  for  basic  F-16C/D  data.  Both  systems  were  able  to  provide  most  of  the 
data  we  wanted.  REMIS  supplied  only  on-equipment  not  total,  man-hours.  TICARRS 
supplied  two  estimates  of  sorties,  one  based  on  debrief  records  that  was  similar  to  the 
REMIS  number,  and  a  second  higher  number  based  on  utilization  reports,  that  users  say  is 
more  accurate.  There  were  some  major  differences  in  the  numbers. 

In  summary,  in  several  cases,  the  REMIS  data  provided  were  for  a  different  time 
period  than  requested,  which  prevented  us  from  comparing  the  REMIS  data  with  the 
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TICARRS  data.  In  one  instance,  I  ICARRS  provided  data  for  U.S.  F-lSs  only  rather  than 
worldwide  F-15s,  so  we  could  not  compare  the  data  directly.  REMIS  provided  some  C- 
141  data  not  relevant  to  our  request.  (REMIS  output  on  the  C-141  is  obtained  through  an 
interface  with  G081;  the  C-141  fleet  does  not  use  CAMS.) 

With  regard  to  ease  of  use,  the  difficulty  of  pulling  data  from  the  three  subsystems 
of  REMIS  was  obvious.  Developers  of  the  different  subsystems  of  REMIS  within  LCS 
had  to  confer  to  determine  which  subsystem  was  the  best  one  to  use. 

Funhermore,  the  people  who  retrieved  the  data  for  our  test  were  experts.  An 
average  user  would  probably  encounter  even  more  difficulty.  Aggregation — the  ability  to 
get  a  single  bottom-line  total  without  all  the  components — is  difficult  in  the  canned  queries 
in  EIMSURS  and  PPS.  In  many  cases,  REMIST ALK,  the  customized  inquiry  utility,  was 
the  only  way  to  retrieve  aggregate  data  without  a  lot  of  extraneous  output,  and 
REMISTALK  response  times  are  considerably  longer  than  those  for  standard  queries. 

The  TICARRS  data  was  retrieved  by  DRC  representatives  more  easily  and 
promptly  than  the  REMIS  data  could  be  retrieved  by  REMIS  developers.  The  TICARRS 
data  were  provided  on  letter-sized  laser-printed  paper  and  included  a  roadmap  through  the 
data.  The  REMIS  response  was  on  wide-carriage  paper  and  was  very  difficult  to  follow. 

2.  TICARRS 

TICARRS’s  central  data  base  design  provides  the  advantage  that  most  of  the  data 
need  to  be  handled  only  once,  the  initial  entry  into  the  TICARRS  system.  There  is  no 
question  of  inconsistent  edits,  as  there  is  with  CAMS  and  REMIS.  Under  TICARRS  direct 
entry,  errors  are  allowed  to  persist  only  long  enough  to  allow  flight  line  maintainers  to  do 
their  work.  All  errors  have  to  be  corrected  or  addressed  through  supervisor  review  before  a 
job  can  be  closed  out. 

Lockheed  Corporation  has  performed  periodic  physical  inventories  of  serially- 
controlled  components  on  F-16s  during  phase  inspections  at  the  388th  Fighter  Wing  and 
compared  them  with  the  data  in  the  maintenance  data  systems.  These  data  have  been 
collected  from  June  1985  to  the  present  and  provide  an  indication  of  data  accuracy  for  both 
direct  entry  TICARRS  and  CAMS.  Figure  V-10  shows  the  results  in  terms  of  the 
percentage  of  changes  from  one  periodic  inventory  to  the  next  that  were  not  recorded  in  the 
data  system.  Under  direct-entry  TICARRS,  incorrect  data  decreased  steadily  over  time  to 
an  error  rate  of  about  6  percent  With  CAMS,  missing  data  rates  were  considerably  higher. 
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with  the  latest  rate  being  25  percent.  It  is  not  clear  from  existing  data  whether  the  trend  is 
showing  a  decline  or  remaining  stable  within  the  20-  to  40- percent  range. 


The  Operational  Assessment  at  Seymour  Johnson  AFB  identified  a  number  of 
instances  where  apparent  software  deficiencies  in  TICARRS  led  to  problems  of  data 
accuracy  and  validity.  Of  the  149  Difficulty  Reports  (DIREPs)  generated  during  the 
Operational  Assessment,  about  30  percent  (approximately  47)  led  to  data  inaccuracies. 
Many  of  these  were  corrected  during  and  immediately  after  the  Operational  Assessment 
with  minimal  resources. 

3 .  Integrity  and  Security  of  Data  Input 

Another  aspect  of  data  accuracy  and  completeness  is  integrity  and  security.  A 
maintenance  data  system  requires  procedures  to  ensure  that  only  authorized  persons  can 
enter  or  change  data. 

If  security  procedures  are  inadequate,  unauthorized  persons  can  enter  inaccurate 
data  or  change  data.  As  noted  previously,  CAMS  developers  have  acted  to  tighten  security 
in  response  to  an  Air  Force  Audit  Agency  report.  The  current  REMIS  security  procedures 
raise  issues  with  configuration  management. 

REMIS  requires  a  password  before  configuration  data  can  be  entered;  however, 
once  access  is  obtained,  data  can  be  entered  on  any  system.  REMIS  developers  are  aware 
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of  this  difficulty,  but  have  concluded  that  it  is  too  difficult  to  resolve  within  the  current 
REMIS  architecture  and  cost  constraints.  TICARRS  security  allows  segregation  of 
configuration  data  entry  by  system. 

4 .  Summary 

The  available  information  suggests  that  TICARRS  under  direct-entry  provided 
better  and  more  complete  data  than  the  current  CAMS/REMIS  system  attains  in  most  cases. 
A  summary  of  our  assessment  of  the  capability  of  the  systems  to  provide  adequate  data  is 
presented  in  Table  V-13.  This  assessment  is  based  on  the  functions  and  associated  accuracy 
requirements  noted  in  Table  ni-l  of  this  paper. 


Table  V-13.  A  Comparison  of  Data  Accuracy  and  Completeness 


Function 

Acaaacy 

Reguirement 

CAMS/REMIS 

Accuracy 

TICARRS  Direa- 
Entry  Accuracy 

Safety  of  Flight 

Engines 

99% 

99% 

Not  Capable 

Egress 

99% 

Unknown 

92%  to  96% 

Mission-critical  equipment 

95%  to  98% 

68%  to  76% 

92%>  to  96% 

Predictive  R&M  analysis  and 
warranty  tracking 

90%  to  98% 

68%  to  76% 

92%  to  96% 

Wartime  readiness,  flying-hour 
program,  and  werqxHi  syston 
possession  management 

95% 

95% 

95% 

R&M  analysis,  maintenance 
activities  including  production 
scheduling,  and  conflguration 
managonent 

70%  to  90% 

68%  to  76% 

92%  to  96% 

CAMS/REMIS  has  shown  that  it  can  provide  highly  accurate  data  for  engines, 
where  accuracy  is  very  important.  It  also  does  an  adequate  job  in  the  area  of  EIMSURS 
data.  Outside  the  engine  arena,  however,  product  performance  data  appears  to  be  much  less 
accurate  in  CAMS/REMIS  than  in  TICARRS. 

Several  significant  problems  in  CAMS/REMIS  need  to  be  to  fixed  before  it  can 
routinely  provide  reliable  data.  There  has  not  been  sufficient  time  since  die  fix  to  RECFU  n 
to  assess  whether  data  accuracy  has  improved.  After  that.  CAMS  and  REMIS  developers 
will  have  to  tackle  a  large  backlog  of  reported  problems.  In  addition,  CAMS  suffers  from 
inadequate  data  editing  and  data  error  control.  Substantial  effort  must  be  expended  to 
improve  and  enhance  the  CAMS  editing  capability  to  permit  erroneous  data  entry  to  be 
caught  and  corrected  when  it  is  being  entered.  Because  REMIS  has  the  only  fleet-wide 
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view  of  maintenance  data,  this  work  must  be  done  in  conjunction  with  REMIS  to  obtain  an 
effective,  complete,  and  accurate  data-entry  system  and  data  base.  In  particular,  high-speed 
communication  lines  between  CAMS  at  the  RPCs  and  REMIS  will  be  necessary  to  allow 
the  CAMS  editing  to  obtain  a  fleet-wide  view  of  the  configuration,  work  unit  code,  and  part 
and  serial  number  data. 

Direct-entry  TICARRS  has  had  fewer  problems  with  data  accuracy,  perhaps 
because  it  supports  fewer  systems  and  types  of  equipment.  It  is  not  possible  to  say  that 
expansion  of  TICARRS  would  be  problem-free.  Indeed,  the  Seymour  Johnson  experience 
indicates  that  there  would  be  some  problems.  Nevertheless,  the  architecture  of  TICARRS, 
which  would  remain  essentially  the  same  under  expansion,  seems  to  result  in  more 
complete  data,  fewer  errors,  and  more  reliable  error-correction  when  data  is  entered  directly 
rather  than  through  CAMS. 

E.  ADAPTABILITY 

This  section  provides  an  overview  of  the  adaptability  of  CAMS/REMIS  and 
TICARRS  to  the  future  operating  environment  and  condition  of  the  Air  Force.  A  more 
detailed  discussion  is  provided  in  Appendix  E. 

Evaluating  adaptability  of  these  systems  requires: 

•  an  estimate  of  what  the  Air  Force  of  the  future  will  be  like,  and 

•  an  assessment  of  the  capability  of  the  current  systems  to  move  toward  the 
desired  future  environment. 

1 .  The  Future  Air  Force 

The  nature  of  the  future  Air  Force  with  respect  to  factors  that  affect  the  operation 
and  effectiveness  of  maintenance  information  systems  depends  on  evolution  in  two  areas: 
the  operational  policies  and  practices  of  the  Air  Force  and  technology.  Over  the  time  period 
of  interest  (the  next  seven  to  ten  years)  policies  and  practices  can  be  expected  to  change 
because  of  the  changed  nature  of  the  global  military  environment  and  because  of  the 
pressure  that  tight  budgets  place  on  the  Air  Force  to  Hnd  new  efHciencies.  Technology  can 
be  expected  to  change  in  weapon  system  design,  in  equipment  fault  diagnosis  and 
maintenance,  and  in  the  features  and  portability  of  information  systems.  Since  change  in  the 
Air  Force  does  not  occur  overnight,  our  expectations  are  heavily  weighted  by  current 
Department  of  Defense  and  Air  Force  initiatives. 


V-63 


Some  key  features  of  the  expected  future  environment  are  as  follows: 

•  rapid  deployment  of  ad  hoc  force  elements; 

•  greater  use  of  two-level  maintenance,  placing  a  premium  on  data  completeness 
and  accuracy  for  identification  of  problem  parts  and  on  management  of  the 
repair  and  transportation  process; 

•  relatively  sparse  operations  and  maintenance  budgets  that  will  require  smart  use 
of  accurate  data  to  maintain  readiness; 

•  extensive  built-in  testing  and  status-reporting  capability  in  new  weapon 
systems; 

•  use  of  automated  maintenance  aids  that  incorporate  aspects  of  artificial 
intelligence  will  use  maintenance  data  to  aid  in  fault  diagnoses;  and 

•  user-friendly  information  systems  that  minimize  the  need  for  data  entry. 

In  light  of  these  expected  developments,  the  weapons  maintenance  information 
system  of  the  future  must  be  capable  of  providing  the  following  strategically  important 
features: 

•  be  deployable  at  the  squadron  level, 

•  be  adaptable  to  new  weapon  systems,  maintenance  technology,  tools  and 
procedures, 

•  be  able  to  interface  with  other  systems, 

•  be  able  to  adapt  to  Air  Force  organizational  changes  and  processes, 

•  provide  accurate  and  timely  data, 

•  take  advantage  of  emerging  information  system  technology  to  provide  user- 
friendly  (user  productive  and  efficient)  data  collection  and  data  access, 

•  permit  access  to  centralized  data  for  a  fleet-wide  perspective  of  status, 
reliability,  and  maintainability  data,  and 

•  not  be  too  expensive. 

We  envision  a  maintenance  information  system  that  is  oriented  around  squadron- 
level  modules  that  use  local  area  networks  (LANs)  and  client-server  data  base  systems 
(CSDSs).  The  squadron-level  data  base  has  access  to  and  may  be  accessed  both  by  the 
wing  CSDS  and  by  a  central  data  repository  for  all  Air  Force  weapon  systems.  Deployed 
squadrons  also  use  the  CSDS-LAN  system  configuration,  but  arc  able  to  sustain  operation 
without  continuous  communication  to  the  central  repository.  The  deployed  squadron 
periodically  uses  satellite  links  to  communicate  essential  data  to  and  from  the  central 
repository. 


V-64 


At  the  squadron  level,  LANs  are  used  to  communicate  with  a  variety  of  work 
stations  and  test  stations,  all  of  which  use  a  standard  LAN  adapter  and  common 
serial/parallel  interface.  Test  equipment,  smart  terminals,  or  workstations,  which  arc  linked 
to  hand-held  and  integrated  data  collection  devices  (IMIS  or  F-22  AIMS),  are  interfaced 
with  the  LAN  to  send  data  to  the  CSDS.  This  interface  is  facilitated  by  using  standard 
interfaces  and  common  communication  protocols.  Modem  software  programs  that  use 
Windows-type  techniques  allow  personnel  entering  data  to  select  from  lists  rather  than  to 
enter  the  data  manually,  and  smart  personal  computer  terminals  provide  graphics  displays 
and  voice  response  units  to  allow  critical  data  to  be  analyzed  and  to  help  in  training  new 
personnel. 

2.  Capability  of  Current  Systems  to  Move  Toward  the  Future  Environment 

The  foregoing  vision  of  how  the  Air  Force’s  maintenance  information  system  needs 
to  evolve  was  developed  by  IDA.  It  was  based  on  information  received  from  the 
CAMS/REMIS  Program  Office,  Dynamics  Research  Corporation,  the  F-22  System 
Program  Office,  and  other  government  and  industry  sources.  The  technological  path  is  not 
difficult  to  visualize;  however,  the  right  way  to  adapt  to  it  in  the  Air  Force  environment  is 
not  so  easy  to  envision.  It  is  unlikely  that  the  kind  of  information  system  the  Air  Force 
needs  can  be  designed  and  put  in  place  simply.  Current  systems  and  procedures  must  be 
dealt  with,  and  such  systems  are  not  replaced  quickly.  Selection  of  technologies  and 
products,  hardware  and  software,  must  be  such  that  they  can  be  evolved  into  an 
information  system  that  takes  full  advantage  of  the  emerging  technologies  to  meet  the  needs 
of  the  Air  Force. 

The  existing  weapons  information  systems,  CAMS/REMIS  and  TICARRS,  are  the 
starting  points  for  the  information  system  technological  path.  Does  one  of  them  stand  out 
from  the  other  as  the  obvious  choice  to  serve  as  the  foundation  for  evolving  to  the  system 
of  the  future? 

a.  CAMS/REMIS 

CAMS  would  no  longer  be  necessary  in  the  type  of  system  described  above.  Once 
the  squadron  CSDSs  were  established,  base  by  base,  and  interfaced  directly  with  REMIS, 
CAMS  would  simply  be  replaced,  functionally  and  physically.  CAMS  does  not  have  the 
function  or  structure  to  perform  well  in  a  client-server  environment.  CAMS  hardware  and 
software  are  not  built  to  allow  users  to  operate  it  independently  of  the  mainframe  system. 
The  architecture  is  not  well-suited  to  support  the  movement  away  from  mainframes  to  high- 
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speed,  high-capability  networks  of  computers  and  communications  equipment.  In  addition, 
the  wing-oriented  nature  of  CAMS  makes  the  development  of  data  bases  tailored  for  rapidly 
assembled  deploying  squadrons  very  difficult. 

REMIS  has  several  architectural  features  that  will  be  required  by  a  central 
repository.  First,  it  has  a  data  base  that  is  designed  to  contain  fleet-wide  data.  Second,  it 
has  a  very  good  (but  inefficient)  network  management  program,  and  it  has  the  mechanisms 
needed  to  download  data  and  programs  to  CSDSs.  It  also  has  a  capability  to  interface  with 
other  systems.  However,  the  REMIS  implementation  of  its  data  report  mechanism  may  be  a 
continuing  problem.  A  key  concern  is  that  the  REMIS  data  base  structure  (relational)  may 
not  be  well-suited  to  provide  expeditious  central  data  reporting  capability.  REMIS  is  having 
trouble  with  this  issue  now.  It  may  be  intrinsic  to  the  system  architecture. 

b.  TICARRS 

The  TICARRS  system  architecture  allows  relatively  rapid  addition  of  new  weapon 
systems  and  equipment.  It  also  has  a  mechanism  for  bulk  input/output,  which  could  be  the 
basis  for  interfaces  with  other  systems  (as  it  is  now  with  CAMS).  TICARRS  currently 
supports  a  LAN  system  for  the  Maintenance  Operations  Center  (MOC),  which  could  be  the 
foundation  for  more  sophisticated  support  Tbe  TICARRS  Conversation  Manager  software 
would  not  be  required  to  support  a  squadron  CSDS.  TICARRS  uses  a  network  type  data 
base,  which  is  suited  to  handling  many  data  base  activities  rapidly.  While  TICARRS 
performance  was  satisfactory  when  supporting  the  F- 16  fleet  with  direct  input  it  remains  to 
be  seen  how  it  would  measure  up  in  a  full  fleet-wide  support  environment 

Neither  REMIS  nor  TICARRS  has  a  distinct  technical  advantage  relative  to  each 
other  in  terms  of  its  ability  to  serve  as  the  basis  for  the  future  system.  TICARRS  has 
demonstrated  a  capability  to  rapidly  and  efficiently  change  its  application  software  to  meet 
new  requirements;  however,  in  terms  of  the  next-generation  maintenance  information 
system  requirements,  it  possesses  a  slight  technical  advantage  over  REMIS  (due  to 
REMIS ’s  lower  processing  speed),  but  clearly  not  sufficient  to  weigh  heavily  in  its  favor. 
While  parts  of  each  system  might  be  useful  in  future  development  efforts,  we  believe  that 
both  are  likely  to  be  overtaken  by  emerging  technologies  and  trends  in  information 
processing. 
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F.  LOGISTICS  AND  OPERATIONAL  EFFECTIVENESS 


1 .  Objective 

This  section  of  the  report  describes  the  effect  (or  lack  of  effect)  of  CAMS/REMIS 
and  TICARRS  on  maintenance-related  support  and  weapon  system  performance.  The 
objective  was  to  determine  if  it  can  be  demonstrated  that  the  information  systems 
significantly  affect  the  logistics  support  and  operational  environment  The  analysis  focuses 
on  short-term  effects  from  changes  to  information  system  support  of  the  weapon  systems. 

A  comparison  is  made  of  support  and  performance  over  the  period  of  time  when 
TICARRS  users  were  required  to  switch  from  direct  entry  into  TICARRS  to  entry  through 
CAMS  and  then  into  TICARRS.  Weapon  systems  supported  by  TICARRS  are  the  focus 
here,  but  two  non-TICARRS  weapon  systems  are  included  to  control  for  any  simultaneous 
changes  in  Air  Force  activities  that  might  obscure  the  true  effects  of  the  information 
systems. 

2 .  Approach 

The  approach  was  to  use  a  time  series  of  weapon  system  and  logistics  performance 
data  for  tactical  weapon  systems  to  create  a  historical  snapshot  from  which  the  conversion 
from  direct-entry  TICARRS  reporting  to  CAMS-entry  TICARRS  reporting  can  be 
evaluated.  The  CAMS  conversion  dates  were  then  “overlaid”  onto  these  trends  to  determine 
if  there  were  significant  differences  in  pre-  and  post-CAMS  conversion  performance. 
Formal  statistical  tests  were  used  to  evaluate  these  time  trends. 

Data  requests  were  made  of  DRC  and  the  CAMS/REMIS  Program  Management 
Office  (PMO)  to  provide  the  following  information: 

•  inventory,  status,  and  utilization  data  for  tactical  weapon  systems  over  the 
period  1982  through  1992  and 

•  product  performance  data,  including  failures,  demands,  and  maintenance  man¬ 
hours,  by  two-digit  work  unit  code  for  the  same  weapon  systems. 

DRC  provided  these  data  for  all  F-16  and  F-15  weapon  systems  on  the  TICARRS 
system.  The  CAMS/REMIS  PMO  was  unable  to  provide  all  of  the  information  requested, 
and  limited  its  response  to  providing  weapon  system  performance  data  only,  for  F-1 11s, 
F-4Es,  and  other  weapon  systems  for  the  recent  past.  A  similar  request  to  ACC  yielded 
more  historical  data  for  non-TICARRS  weapon  systems. 
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Weapon  system  performance  was  measured  by  the  mission-capable  (MC)  rate.  The 
MC  rate  is  the  fraction  of  possessed  hours  that  aircraft  are  either  fully  mission  capable  or 
partially  mission  capable.  The  hypothesis  tested  was  that  wings  had  higher  MC  rates  with 
one  of  the  information  systems  than  with  the  other.  This  would  reflect  well  on  the 
information  system  associated  with  higher  aircraft  availability. 

If  a  better  information  system  enhances  aircraft  availability,  this  is  presumably  the 
result  of  either  parts  breaking  less  (perhaps  because  problem  parts  are  eliminated  from  the 
system)  or  being  fixed  more  quickly  (or  both).  Mean  time  between  failures  (MTBF)  and 
maintenance  man-hours  per  flying  hour  (MMH/FH)  were  examined  as  a  function  of  the 
information  system  in  use.  The  analyses  concentrated  on  fire-control,  which  includes  radar 
and  inertial  navigation  system  equipment  (WUC  74). 

The  hypotheses  that  the  information  system  being  used  could  be  linked  to  MC  rates, 
MTBF,  and  MMH/FH  were  tested  in  two  ways.  The  first  focused  solely  on  the  F-16 
weapon  system  and  its  conversion  from  direct-entry  TICARRS  to  CAMS-TICARRS,  entry 
which  occurred  during  the  period  from  late  1988  through  1989.  Time  trends  not  associated 
with  the  conversion  from  TICARRS  to  CAMS  were  corrected. 

The  second  test  focused  on  three  weapon  systems — ^F-16,  F-111,  and  F-4E — all  of 
which  had  increasing  or  level  MC  rates  over  the  relevant  period  of  this  evaluation.  The 
analysis  examined  how  the  MC  rate  changed  from  the  period  before  CAMS  conversion  for 
the  F-16  began  to  the  period  after  it  was  completed.  If  CAMS  helped,  one  would  have 
expected  the  F-16  to  have  improved  significantly  more  between  the  two  periods.  A  finding 
that  the  F-16  improved  less  would  have  indicated  that  TICARRS  supported  wing 
operations  and  maintenance  better  than  CAMS. 

A  detailed  discussion  of  the  results  of  our  statistical  analyses  is  provided  in 
Appendix  F.  The  results  are  summarized  below. 

3.  Results 

The  analyses  consistently  indicated  that  the  conversion  to  CAMS-entry  from  direct- 
entry  TICARRS  for  the  F-16  weapon  system  produced  no  significant  change  in  weapon 
system  performance.  Compared  with  the  F-111  and  F-4E,  the  F-16  exhibited  no 
differences  in  MC  rates  during  the  period  of  conversion  to  CAMS  entry.  Regarding  product 
performance  of  a  selected  WUC  for  the  F-16  aircraft,  the  results  indicate  that  the 
conversion  itself  did  not  affect  MTBF  or  MMH/FH,  although  some  time-sensitive 
variations  occurred  during  die  conversion  period. 
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These  results  do  not  necessarily  mean  that  the  choice  of  system  is  not  important  to 
logistics  performance  or  aircraft  readiness.  The  analysis  presented  above  is.  by  design, 
limited  to  short-term  impacts  of  the  information  systems  on  logistics  and  weapon  system 
performance  measures.  Impacts  that  are  visible  only  over  a  longer  time  period  must  not  be 
ignored.  An  important  reason  for  wanting  maintenance  data  is  to  facilitate  identification  of 
problem  parts  and  potentially  worthwhile  modifications.  The  value  of  data  in  these  roles 
would  only  be  evident  over  a  longer  period  of  time,  and  would  be  unlikely  to  be  identified 
in  the  analyses  discussed  above.  Just  as  important,  the  fact  that  operational  effectiveness 
under  the  two  systems  is  essentially  equal  does  not  necessarily  mean  that  the  systems  make 
no  difference.  Wing  personnel  may  find  technical  work-arounds  to  overcome  problems 
with  the  information  systems,  or  the  Air  Force  may  maintain  readiness  by  using  more 
resources  (like  buying  more  spares  or  working  longer  hours).  Unfortunately,  these  kinds 
of  adaptations  are  difficult  to  detect.  The  analysis  presented  here  should  not  be  taken  to 
mean  that  better  data  have  no  payoff  to  the  Air  Force. 

G.  SUMMARY 

This  chapter  has  provided  a  great  deal  of  detail  about  the  effectiveness  of 
CAMS/REMIS  and  TICARRS.  It  lays  the  foundation  for  our  definition  and  estimation  of 
viable  alternatives  to  the  current  maintenance  data  support  environment.  Chapter  VI 
presents  the  alternatives  and  Chapter  VII  provides  the  cost  calculations  for  those 
alternatives. 

A  summary  of  the  capabilities  and  deficiencies  of  CAMS/REMIS  and  TICARRS  is 
presented  in  Table  V-14. 


Table  V-'14.  Summary  of  Effectiveness  Analysis 


Dimension  of  Effectiveness 

Conclusion 

Functionality 

Greater  for  CAMS/REMIS 

Scope 

Operating  diaracteristics 

Greater  for  CAMS/REMIS 

Availability 

Better  for  TICARRS 

Respcmsiveness 

Better  for  TICARRS 

Ease  of  use 

Equal  at  wing  level,  TICARRS  better  elsewliere 

Data  accuracy  and  completeness 

Better  for  TICARRS 

Adaptability 

TICARRS  better  in  short-term;  little  long-term 
difference 

Logistics  and  operational  effectiveness 

No  difference  found 
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The  most  important  differences  in  system  effectiveness  are  that  CAMS/REMIS  has 
greater  functionality  and  much  greater  scope  than  TICARRS,  but  TICARRS  has  better 
operating  characteristics  and,  more  important,  has  demonsu-ated  a  greater  ability  to  develop 
and  maintain  accurate  data. 
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VI.  DEFINITION  OF  ALTERNATIVES 


This  report  has  provided  evidence  that  both  CAMS/REMIS  and  TICARRS  have 
limitations  that  should  not  be  present  in  an  Air  Force  standard  maintenance  system.  This 
chapter  outlines  enhancements  that  should  be  made  and  areas  that  must  be  addressed  in 
each  system.  The  enhancements  are  presented  in  the  context  of  two  alternative  scenarios 
regarding  the  future  of  these  systems: 

•  Alternative  1:  an  enhanced  version  of  CAMS/REMIS  that  incorporates 
improvements  that  are  either  under  way  or  planned  for  the  near  future.  With 
this  alternative,  CAMS/REMIS,  with  needed  enhancements,  continues  as  the 
Air  Force’s  standard  system.  TICARRS  is  retired,  though  it  continues  to 
operate  until  REMIS  teaches  full  operational  capability. 

•  Alternative  2:  an  enhanced  version  of  TICARRS  92  that  is  expanded  in 
function  and  scope  to  allow  it  to  perfoim  all  the  tasks  it  must  perform  to  replace 
CAMS/REMIS.  TICARRS  becomes  the  Air  Force  standard  system. 
CAMS/REMIS  is  retired.  Because  of  the  length  of  time  it  will  take  to  complete 
aU  the  functional  and  administrative  steps  necessary  to  make  TICARRS  the  Air 
Force  standard  maintenance  information  system,  the  development  of  REMIS 
will  proceed  to  full  operational  capability.  The  Generic  Configuration  Status 
Accounting  System  (GCSAS)  will  be  deployed,  and  the  information  systems 
that  REMIS  is  designed  to  supplant  (except  for  TICARRS)  will  be  turned  off. 
Some  investment  will  be  made  to  improve  the  performance  of  REMIS,  and  the 
costs  involved  in  developing  accurate  configuration  information  for  GCSAS 
will  be  incurred.  CAMS  efforts  pertaining  to  Computer  System  Requirements 
Document  (CSRDs)  will  continue.  Other  development  efforts  related  to  CAMS 
will  cease. 

Each  of  these  alternatives  is  explored  in  the  subsections  that  follow. 

Regardless  of  which  alternative  is  pursued,  we  assume  that  the  system  not  chosen 
will  continue  to  operate  until  the  system  that  is  chosen  reaches  full  operating  capability.  At 
that  point,  the  systems  would  exhibit  roughly  equivalent  functionality,  scope,  operating 
characteristics,  and  so  on.  Chapter  Vn  contains  estimates  of  the  costs  to  implement  these 
enhancements  and  includes  costs  of  the  transition  to  a  single  system. 
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A .  ALTERNATIVE  1:  ENHANCED  VERSION  OF  CAMS/REMIS 


1 .  Areas  of  Enhancement  for  CAMS 

As  was  discussed  in  Chapter  V,  CAMS  suffers  from  poor  availability  and  poor 
transaction  response  time.  Unfortunately,  it  is  unclear  how  much  of  the  problem  is  tied  to 
CAMS  software  and  how  much  is  due  to  inadequate  base-level  operations.  The  evidence 
indicates  that  CAMS  has  good  performance  and  good  availability  at  a  few  bases,  leading  to 
the  conclusion  that  the  base-level  operation  must  be  a  major  contributor  to  the  problem.  The 
move  to  Regional  Processing  Centers  (RPCs)  is  expected  to  improve  the  situation.  Even 
so,  CAMS  software  reliability  must  be  examined  on  a  continuing  basis  to  ensure  that  it  is 
not  contributing  to  availability  problems.  This  must  be  an  ongoing  effort  of  the  CAMS 
maintenance  staff. 

With  any  large  data  system,  there  is  a  continual  need  for  human  and  dollar 
resources  to  correct  problems  and  defects  and  improve  system  performance.  CAMS  is  no 
different,  but  it  suffers  from  a  long  backlog  of  reported  but  unresolved  problems.  In  the 
past,  priority  has  been  placed  on  developing  the  increments  of  new  capability  rather  than  on 
correcting  problems  or  improving  system  performance. 

An  effort  is  being  made  by  the  current  CAMS  management  to  address  some  of  these 
problems.  A  special  $1  million  optimization  effort  is  ongoing  with  contract  assistance  from 
the  Harris  Corporation.  This  effort  is  limited  to  useability  problems  (e.g.,  use  of  multiple 
screens,  improved  error  messages,  etc.).  In  addition,  there  is  currently  a  30-staff-year 
backlog  of  approved  functional  modifications  (CSRDs)  to  CAMS.  In  the  cost  analysis 
described  in  Chapter  Vn,  we  included  personnel  to  address  these  kinds  of  software 
maintenance  problems. 

CAMS  also  suffers  from  inadequate  data  editing  and  control.  Substantial  effort 
must  be  expended  to  improve  and  enhance  CAMS’s  editing  capability  to  permit  erroneous 
data  entry  to  be  identified  and  corrected  when  data  are  entered.  Because  REMIS  has  the 
only  fleet-wide  view  of  maintenance  data,  this  work  roust  be  done  in  conjunction  with 
REMIS  to  obtain  an  effective,  complete,  and  accurate  data-entry  system  and  data  base.  In 
particular,  high-speed  communication  lines  between  CAMS  at  the  RPCs  and  REMIS  will 
be  necessary  to  allow  the  CAMS  editing  to  obtain  a  fleet-wide  view  of  the  configuration, 
woik  unit  code,  part  number,  and  serial  number  data.  The  editing  enhancement  would 
totally  replace  the  current  process  of  REMIS  responding  to  the  individual  bases  with  error 
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messages  an  hour  or  two  after  the  fact,  with  the  expectation  of  error  correction  by  the 
flight-line  personnel  or  the  CAMS  data  base  manager. 

2.  Areas  of  Enhancement  for  REMIS 

Several  key  REMIS  performance  issues  must  be  addressed.  Although  response 
time  is  adequate  for  simple  transactions,  it  is  very  slow  for  users  requesting  reports, 
especially  ad  hoc  reports.  The  following  areas  should  be  addressed  to  speed  up  the  REMIS 
report-generation; 

•  Balance  the  central  computer  workload.  As  discussed  in  Chapter  V,  most  of 
the  load  is  currently  on  HQl. 

•  Reduce  the  amount  of  computer  resources  used.  The  programs  that  make  up 
the  network  control  system  (operations  overhead)  and  those  that  process  input 
from  CAMS  (system-to-system  input)  will  need  to  be  redesigned  and  recoded. 

•  Add  computing  power.  Additional  computing  power  will  be  necessary  to 
handle  the  full  workload  expected  by  REMIS. 

•  Reduce  the  time  required  to  access  data  through  REMIST ALK. 

3 .  Transition  Schedule 

TICARRS  would  be  phased  out  beginning  when  REMIS  reaches  full  operational 
capability  (projected  for  January  1995).  Since  the  weapon  systems  supported  by  TICARRS 
are  also  supported  by  REMIS,  we  assume  that  the  phasing  out  could  be  done  quickly.  For 
the  purposes  of  the  analysis,  we  assumed  this  would  be  done  over  a  three-month  period 
(January  through  March  1995).  Thus,  TICARRS  would  operate  throughout  FY  1994  and 
through  the  first  six  months  of  FY  1995.  The  process  of  developing  improvements  to 
CAMS  and  REMIS  would  continue  until  the  end  of  FY  1996,  and  operational  test  and 
evaluation  would  occur  in  FY  1997, 

B .  ALTERNATIVE  2;  ENHANCED  VERSION  OF  TICARRS 

1 .  Areas  of  Enhancement  for  TICARRS 

If  TICARRS  is  to  become  the  standard  Air  Force  maintenance  system,  it  must  be 
expanded  to  cover  all  weapon  systems  cunently  covered  by  CAMS/REMIS.  TICARRS 
would  have  to  be  activated  to  support  89  Air  Force  bases.  The  following  major  steps  must 
be  taken  to  accomplish  this: 

•  Additional  hardware  must  be  purchased  in  order  to  handle  the  greatly  increased 
demands  on  processing  and  data  storage. 
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•  The  application  software  must  be  enhanced.  Eight  enhancements  have  been 
identified  as  necessary  to  give  TICARRS  the  required  functionality.  They  are; 

-  Comprehensive  Engine  Management  System  (CEMS)  interface, 

-  Standard  Base  Supply  System  (SBSS)  interface, 
aerospace  ground  equipment, 

-  production  scheduling  (comparable  to  CAMS  380  screen), 
maintenance  snapshot  (comparable  to  CAMS  122  screen), 
automated  aircraft  forms, 

-  personnel  and  training,  and 

Product  (Quality  Deficiency  Report  (PQDR). 

•  The  data  base  must  be  initialized  for  each  of  44  weapon  systems,  simulators 
and  trainers,  tactical  missiles,  and  communications-electronics  equipment 

•  Communications  must  be  added  from  the  bases  to  TICARRS  to  allow  direct 
input 

•  A  total  of  254  units  must  be  activated  and  users  trained.  This  estimate  takes  the 
expected  reduction  in  the  size  of  the  Air  Force  into  account 

Estimates  of  the  resources  required  for  each  of  these  steps  are  discussed  in  Chapter  VII. 

2 .  Transition  Schedule 

We  assumed  that,  before  TICARRS  can  become  the  standard  Air  Force 
maintenance  data  system,  it  must  proceed  through  a  complex  acquisition  review  process. 
Five  major  activities  must  be  completed  before  TICARRS  could  be  considered  fully 
deployed: 

•  contract  award, 

•  system  development 

•  operational  test  and  evaluation, 

•  Major  Automated  Information  System  Review  Council  (MAISRC)  review,  and 

•  deployment 

Figure  VI- 1  shows  the  development  approval,  and  deployment  schedule  that  has 
been  used  in  developing  the  costs  that  are  {resented  in  Ch^ter  Vn. 
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Despite  the  fact  that  DRC  developed  TICARRS  and  hat,  been  operating  and 
maintaining  it  for  years.  Department  of  Defense  regulations  specify  that  a  formal  selection 
process  must  be  completed  before  TICARRS  can  be  enhanced.  This  could  be  done  through 
either  a  sole-source  justification  or  a  competitive  procurement.  The  Air  Force  believes  that 
this  process  is  likely  to  take  about  a  year.  If  a  decision  is  made  in  October  1993  to  go  with 
Alternative  2,  the  contract  could  be  awarded  around  1  November  1994.  After  that,  the  other 
scheduled  activities  could  proceed  to  full  operational  capability. 

Modifications  to  the  system  must  be  designed,  and  the  design  must  be  approved. 
Coding  can  begin  before  design  is  complete.  The  software  must  be  tested.  Documentation 
can  be  prepared  during  the  coding  and  contractor  testing  period. 

A  data  base  structure  must  be  developed  for  the  aircraft  (or  other  systems)  to  be 
used  in  the  operational  test  and  evaluation  (OT&E).  The  OT&E  itself  is  expected  to  take 
four  months,  followed  by  two  months  of  analysis  and  report  preparation.  System 
development  and  preparation  for  MAISRC  review  could  be  expected  to  take  16  months 
from  the  date  of  contract  award.  This  would  lead  to  MAISRC  approval  in  March  1996.  At 
that  time,  deployment  of  TICARRS  to  operating  bases  would  begin. 

We  estimate  that  deployment  to  the  254  units  expected  to  be  in  service  at  the  time 
could  be  completed  in  19  months.  As  locations  are  converted  to  TICARRS,  CAMS  would 
be  shut  off.  REMIS  would  be  terminated  when  the  deployment  is  complete,  at  the  start  of 
FY  1998. 

Delays  in  the  acquisition  process,  such  as  might  be  caused  by  contracting  protests 
or  slippage  in  the  development  and  approval  process,  could  push  the  completion  of 
deployment  deep  into  FY  1998.  Chapter  VII  includes  sensitivity  analysis  to  account  for 
these  potential  delays.  On  the  other  hand,  progress  in  the  ongoing  efforts  to  simplify  the 
acquisition  process  could  speed  things  up.  The  cost  implications  of  delays  to  the  process 
are  investigated  as  an  excursion  in  Chapter  VII. 

C.  RISK  ASSESSMENT 

The  two  alternatives  contain  a  number  of  important  changes  to  the  existing  Air 
Force  maintenance  information  systems.  As  with  any  developmental  effort,  there  is  risk.  In 
this  assessment,  the  risk  involves  the  systems  not  being  able  to  perform  as  well  as  desired 
after  spending  the  specified  amount  of  money  on  development 

Table  VI- 1  assesses  the  riskiness  of  the  two  alternatives  with  respect  to  the  most 
critical  elements  of  effectiveness  discussed  in  Chapter  V  and  estimated  in  Chapta  VIL 
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Table  VI-1.  Assessment  of  Technical  Risk  Associated  With  Alternatives 


Aspect  of  Effectiveness 

Alternative  1 

Alternative  2 

Functionality 

None 

Low 

Scope 

None 

Low 

Operating  Characteristics 

Low-Medium 

Low 

Availability 

Medium 

Low 

Response  Time 

Medium 

Low 

F=>se  of  Use 

Low 

Low 

Data  Accuracy  and  Completeness 

Medium 

Low 

Overall  risk  is  low  to  medium  for  Alternative  1  and  low  for  Alternative  2. 

The  move  to  Regional  Processing  Centers  will  not,  by  itself,  improve  the 
availability  of  CAMS.  Improvements  to  availability  are  dependent  upon:  (1)  improved 
software  quality  and  (2)  the  provision  of  the  appropriate  mix  of  skilled  personnel  (data  base 
managers)  at  the  bases. 

Regarding  CAMS/REMIS  response  lime,  it  should  be  possible  to  adequately 
improve  the  responsiveness  of  REMIS,  but  there  is  more  risk  involved  because  of  the 
architecture  of  the  system. 

The  biggest  potential  problem  involves  the  completeness  and  accuracy  of  CAMS 
and  REMIS  data.  The  completion  of  GCSAS  might  improve  the  situation,  but  the  complex 
nature  of  the  interface  between  the  two  systems,  and  the  data  rejects  that  may  still  result, 
could  result  in  data  that  are  not  accurate  enough  to  meet  some  of  the  Air  Force’s 
requirements  (as  described  in  Table  V-13). 

Even  though  deficiencies  of  scope  and  functionality  are  major  weaknesses  of 
TICARRS  today,  the  flexibility  of  the  system’s  architecture  supports  optimism  about 
TICARRS’s  ability  to  be  expanded  as  needed. 
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VII.  ESTIMATING  THE  COSTS  OF  ALTERNATIVES 


This  chapter  contains  the  cost  analysis  for  the  alternatives  defined  in  Chapter  VI.  To 
facilitate  comparison,  the  analyses  follow  a  common  cost  element  structure. 

A.  INTRODUCTION 

First,  we  explain  our  assumptions  concerning  cost,  then  we  present  the  costs  of  the 
individual  information  systems.  Section  C  presents  the  cost  analysis  for  CAMS,  Section  D, 
for  REMIS;  and  Section  E  for  TICARRS.  Section  F  discusses  cost  drivers,  and  Section  G 
presents  the  costs  of  the  two  alternative  approaches  to  meeting  the  Air  Force’s  needs  for 
maintenance  information  that  were  developed  in  Chapter  VI.  Both  of  the  alternatives 
include  cost  components  associated  with  all  three  individual  systems.  Alternative  1  includes 
18  months  of  TICARRS  costs  because  REMIS  is  not  expected  to  be  fully  functional  until 
late  1994.  Alternative  2  includes  four  years  of  CAMS/REMIS  costs  because  we  expect 
enhancement  and  expansion  of  TICARRS  to  take  that  long  if  we  assume  it  undergoes  a 
Major  Automated  Information  System  Review  Council  (MAISRC)  review. 

CAMS,  REMIS,  and  TICARRS  vary  in  terms  of  the  detail  and  visibility  of 
associated  costs.  By  far  the  greatest  amount  of  detail  was  available  for  REMIS.  In 
preparation  for  the  MAISRC  Milestone  HI,  a  Program  Office  Estimate  (POE)  and  an 
Independent  Cost  Estimate  (ICE)  were  prepared.  Detailed  cost  projections  were  also 
available  from  Litton  Computer  Services  (LCS). 

CAMS  costs  proved  to  be  difficult  to  obtain  because  they  are  spread  across  many 
different  organizational  boundaries.  As  previously  stated,  the  Standard  Base  Level 
Computer  (SBLC)  and  the  Regional  Processing  Center  (RPC)  costs  arc  fundamental  cost 
drivers  of  the  CAMS  costs.  Since  the  operation  of  the  SBLCs  are  funded  through  the  Major 
Commands  (MAJCOMs),  there  does  not  appear  to  be  a  single  source  to  obtain  individual 
base-level  SBLC  costs,  or  the  SBLC  costs  attributable  to  CAMS.  Much  of  the  CAMS 
operations  and  maintenance  (O&M)  costs  were  based  on  the  Air  Force  Business  Case 
Analysis  supporting  the  DMRD  924  initiative  and  data  provided  by  the  DMRD  924 
Program  Management  Office  (PMO)  at  Gunter  Standard  Systems  Center  (SSC). 
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TICARRS  had  adequate  documentation  of  current  and  historical  costs.  The  major 
issues  for  this  system  concern  the  costs  to  expand  the  functionality  of  TICARRS  and  to 
transition  254  units  at  89  bases  from  CAMS  to  TICARRS. 

B.  ASSUMPTIONS 

This  section  outlines  the  general  assumptions  we  made  concerning  the  systems 
being  estimated.  For  all  the  cost  estimates,  we  used  FY  1994  constant  dollars. 

1 .  Contractor  Labor 

We  assumed  1,920  hours  per  staff-year  for  contractor  labor.  The  hourly  rate 
assumed  for  software  and  data  base  design  and  implementation  is  $60  per  hour  burdened 
plus  an  additional  15  percent  for  general  and  administration  (G&A)  expenses  plus  fee  for  a 
total  hourly  rate  of  $69.  Multiplying  by  1,920  hours  per  year  gives  an  annual  rate  of 
$132,480.  The  burdened  hourly  rate  assumed  for  computer  operators  is  $35  plus  an 
additional  15  percent  for  G&A  plus  fee  for  a  total  hourly  rate  of  $40.25  (or  $77,280  per 
year).  Those  rates  apply  to  all  contractor  labor  costs,  unless  otherwise  noted. 

2.  Air  Force  Military  and  Civilian  Personnel 

Air  Force  Regulation  (AFR)  173-13,  Attachment  32,  Table  A32-1,  lists  the  pay  for 
the  military  grades,  and  Attachment  31,  Table  31-1,  lists  the  civilian  paygrades.  Unless 
otherwise  noted,  the  paygrades  used  were  GS-09  at  $49,885  for  civilian  and  E-6  at 
$48,622  for  military  employees.  The  amounts  indicated  are  the  Accelerated  Annual  Pay 
Rate  for  FY  1993  based  on  the  FY  1994  President’s  Budget  and  adjusted  to  FY  1994 
dollars.  Accelerated  annual  pay  for  military  personnel  includes  the  costs  of 
government-furnished  quarters  for  personnel  living  in  family  housing  or  dormitories  and 
not  receiving  Basic  Allowance  for  Quarters  (BAQ)  payments;  the  costs  of  preparing  and 
serving  food  in  the  dining  room;  medical  costs  covered  by  the  O&M  appropriation;  and 
commissary  and  exchange  benefits  subsidized  by  appropriated  funds.  Accelerated  annual 
pay  for  Air  Force  civilian  personnel  includes  funded  and  unfunded  retirement  and  benefits. 
The  Air  Force  civilian  factor  follows  the  Office  of  Management  and  Budget  (0MB)  A-76 
guidelines. 

Contractor  costs  include  certain  overhead  costs  (e.g.,  facilities)  that  are  not  included 
in  the  DoD  military  and  civilian  rates  listed  above.  For  cost  comparison  purposes,  the  DoD 
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rates  were  increased  by  the  amounts  listed  as  non-pay  cost  factors  in  AFR  173,  Attachment 
56,  Table  A56-1.  Those  factors  are  as  follows: 

•  Air  Combat  Command  (ACC) — $6,100  per  year, 

•  U.S.  Air  Forces  Europe  (USAFE) — $14,600  per  year,  and 

•  Pacific  Air  Force  (PACAF) — $9,600  per  year. 

The  installation  support  costs  must  be  added  to  make  costs  for  Air  Force  personnel 
comparable  to  costs  for  contractor  personnel.  TTie  cost  factors  include  base  operations,  base 
communications,  real  property  services,  maintenance  and  repair,  child  care,  and  other  base 
overhead  costs.  This  will  result  in  some  variations  in  the  cost  of  personnel  between 
MAJCOMs.  In  addition,  much  of  the  cost  data  provided  did  not  include  a  break  out  of 
civilian  and  military  personnel.  Where  such  information  was  provided,  it  was  used; 
otherwise,  the  personnel  costs  are  based  on  air  average  of  the  selected  civilian  and  military 
paygrades.  The  actual  values  used  in  the  calculations  may  vary  depending  upon  the 
weighted  mix  of  military  and  civilian  personnel. 

3.  Miscellaneous  Assumptions 

There  are  a  number  of  other  assumptions,  including: 

•  cost  estimates  cover  a  ten-year  period  from  FY  1994  through  FY  2003, 

•  there  are  89  bases  and  254  units  (information  provided  by  the  Air  Force),  and 

•  hardware  will  be  replaced/upgraded  after  seven  years.  Replacement  cost  will  be 
based  on  the  same  number  of  units  but  reduced  in  price  (20  percent  per  year)  as 
a  result  of  advances  in  technology. 

It  should  be  noted  that  these  estimates  do  not  represent  programmatic  costs  and  are 
not  intended  to  serve  as  the  basis  for  congressional  appropriations.  The  estimates  for 
CAMS,  for  example,  cut  across  several  program  boundaries.  Our  objective  was  to  estimate 
all  relevant  costs  associated  with  each  system,  regardless  of  the  particular  organization 
responsible  for  those  costs. 

The  following  sections  discuss  the  derivation  of  the  cost  estimates.  In  the  cost 
tables,  we  have  rounded  to  two  signiHcant  digits.  In  some  cases,  the  numbers  may  not  add 
due  to  rounding  error. 
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C .  COST  ESTIMATE  FOR  CAMS 

The  CAMS  cost  estimate  presented  here  includes  both  the  recurring  and 
nonrecurring  costs  to  the  Air  Force  of  operating  CAMS.  Some  of  these  costs  are  not 
included  in  the  CAMS  budget,  but  rather  are  incurred  by  other  organizations  in  the  Air 
Force  and  are  not  tracked  as  CAMS  expenses.  An  example  of  this  is  the  cost  of  operating 
the  RPCs  and  SBLCs  at  each  base.  A  significant  portion  of  the  SBLC  and  RPC  cost  is  due 
to  CAMS  operation,  but  there  are  no  accounting  data  to  substantiate  what  that  portion  is. 
Similarly,  the  cost  of  military  personnel  is  not  reflected  in  some  of  the  budgets.  The  study 
team  cost  projections  include  estimates  of  the  funds  required  for  these  type  of  expenditures. 

The  CAMS  costs  for  the  years  FY  1994-2003  are  heavily  influenced  by  the  SBLC 
regionalization  initiative  (DMRD  924).  All  of  the  computer  support  for  the  SBLC  bases  in 
the  continental  United  States  (CONUS)  and  the  Air  National  Guard  (ANG)  and  Air  Force 
Reserve  (AFRES)  bases  are  planned  to  have  completed  or  started  the  move  to  RPCs  by  end 
of  calendar  year  1994.  Only  14  out  of  a  total  of  76  bases  will  remain  to  be  completed  by 
mid-year  1995.  In  addition,  the  SBLCs  in  USAFE  and  PACAF  have  plans  to  regionalize 
during  1994.  The  cost  estimate  for  CAMS  assumes  that  all  89  bases  will  be  served  by 
RPCs  in  1994  and  beyond. 

The  RPC  cost  projections  provided  by  the  RPC  PMO  were  in  two  categories, 
(1)  investment  or  nonrecurring  costs  in  budget  program  (BP)  3080  and  (2)  operations  and 
maintenance  or  recurring  costs  in  BP  3400.  This  information  was  a  subset  of  the 
information  provided  by  the  Air  Force  Business  Case  Analysis  supporting  DMRD  924 
(dated  13  December  1991).  In  addition,  discussions  with  the  RPC  PMO  personnel  resulted 
in  information  and  system  usage  data  that  provided  the  basis  for  estimating  the  cost  of  the 
RPC  operation  due  to  CAMS. 

The  personnel  costs  for  the  RPC  organizations  are  based  on  the  identification  of 
military  and  civilian  employees  provided  by  the  CONUS  RPC.  This  yielded  an  average 
cost  of  $55,870  after  the  application  of  the  ACC  non-pay  cost  amount  mentioned 
previously.  PACAF  and  USAFE  costs  were  correspondingly  higher. 

Personnel  costs  were  rated  at  an  average  of  the  civilian  and  military  rates  when  the 
militaiy  and  civilian  employee  numbers  were  not  identified.  The  ACC  amount  was  added  to 
this  average,  resulting  in  a  pay  of  $55,660. 

Table  VII- 1  summarizes  the  costs  estimated  for  CAMS  by  cost  element  The 
individual  cost  elements  are  discussed  in  the  subsections  that  follow.  The  numbers  shown 
in  parentheses  correspond  to  the  numbers  to  the  left  of  the  cost  categories  in  the  table. 
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1.  Nonrecurring  Costs  (1.0) 
a.  Hardware  (1.1) 

The  CAMS  share  of  the  CONUS  Regional  Processing  Centers  hardware  costs  was 
established  by  determining  the  CAMS  utilization  of  the  major  hardware  units  in  the  RPCs. 
Measurement  data  provided  by  the  RPC  PMO  enabled  the  study  team  to  develop  CAMS 
average  hardware  utilization  rates,  which,  in  turn,  were  used  as  the  basis  for  establishing 
the  CAMS  share  of  the  hardware  cost.  Average  CAMS  utilization  for  the  major  hardware 
units  are  shown  in  Table  VII-2. 


Table  Vli-2.  CAMS  Share  of  RPC  Hardware  Costs  (1.11) 


Organization  and  Equipment 

FY  1994  Cost 
(Millions) 

Percentage 

CAMS 

CONUS  RPC  Computers 

$34.3 

17.5% 

CONUS  RPC  Disks 

$3.2 

11.9% 

CONUS  RPC  Silos 

$28.3 

14.8% 

CONUS  Base  Print  Stations 

$7.2 

25.0% 

USAFE  RPC  Upgrades 

$1.6 

15.6% 

PACAF  RPC  Hardware 

$12.5 

16.0% 

The  CAMS  utilization  percentage  was  calculated  by  adding  the  CAMS  share  of 
transaction  processing  utilization,  executive  processing,  batch  processing,  and  demand 
processing  as  determined  by  the  RPC  data.  The  RPC  computer  equipment  has  been 
planned  with  an  excess  capacity  of  25  percent  to  allow  for  continuity  of  operation. 
Therefore,  the  CAMS  utilization  was  increased  by  25  percent  to  account  for  the  spare 
capacity  installed  at  each  RPC. 

b.  Software  (1.2) 

This  element  of  the  cost  estimate  is  concerned  with  the  cost  of  further  development 
of  the  CAMS  application  after  FY  1993.  The  CAMS/REMIS  PMO  does  not  project  any 
funds  for  this  purpose  past  FY  1993,  except  those  funds  provided  by  organizations 
wishing  to  enhance  CAMS  to  support  a  specific  weapon  system  or  activity. 

The  study  team  foresees  future  software  development  costs  in  at  least  the  areas  of 
application  software  and  documentation. 
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(1)  Applications  Software  (1.2.1) 

Approved  CSRDs.  CAMS  has  a  backlog  of  approved  functional 
modifications — Computer  System  Requirements  Documents  (CSRDs) — amounting  to 
approximately  30  staff-years  of  effort  (CAMS  estimate).  This  is  projected  to  extend  over  a 
three-year  period  or  10  staff-years  per  year  for  three  years.  This  work  is  expected  to  be 
completed  by  contractor  personnel  at  the  rates  cited  earlier. 

Enhanced  editing.  The  study  team  believes  that  CAMS  must  be  modified  so  that 
the  data  entry  is  not  only  less  onerous,  but  also  fully  edited,  such  that  erroneous  data  are 
refused  at  the  point  of  entry,  and  at  the  time  of  entry.  This  enhancement  would  totally 
replace  the  current  CAMS  edit  facility  and  the  process  of  REMIS  responding  to  the 
individual  bases  with  error  messages  an  hour  or  two  after  the  fact,  with  the  expectation  of 
error  correction  by  the  users  long  after  they  entered  the  data.  This  is  expected  to  be  an 
extensive  change  to  CAMS,  affecting  20  to  30  percent  of  the  system  program  modules  and 
as  much  as  50,000  lines  of  new  and  changed  code  overall.  The  study  team  estimates  the 
enhanced  data  editing  and  control  would  require  about  15  staff-years  of  development  team 
effort  per  year  for  three  years.  This  enhancement  includes  the  ability  to  communicate 
immediately  with  REMIS  for  data  verification  on  a  fleet-wide  basis  as  necessary.  The  cost 
of  high-speed  communication  lines  has  been  included  in  the  recurring  hardware  cost  to 
accommodate  this  capability.  This  work  is  expected  to  be  completed  by  contractor 
personnel  at  the  rates  cited  earlier. 

System  Integration  and  Test.  This  function  is  currently  performed  by  the  SSC 
QT&E  group  at  Gunter  AFB.  The  costs  projected  for  this  item  are  a  function  of  the 
application  development  effort  The  study  team  estimated  that  5  staff-years  per  year  for  the 
first  three  years  would  cover  the  necessary  effort.  This  not  only  includes  integration  of  the 
CAMS  application  into  the  SBLC  and  RPC  software,  but  includes  distribution  to  the 
various  user  organizations.  This  work  was  projected  at  an  annual  rate  of  $55,870  per  staff 
year. 

OT&E  Support.  These  are  funds  to  support  a  CAMS/REMIS  OT&E.  These 
costs  were  based  on  the  $1.4  million  cost  of  the  GCSAS  OT&E.  The  total  CAMS/REMIS 
cost  was  projected  to  be  $2.5  million,  half  of  which  is  shared  by  CAMS. 

(2)  Documentation  (1.2.3).  This  category  covers  the  cost  of  producing  and 
distributing  user  manuals.  We  estimated  this  as  10  percent  of  the  cost  of  application 
software  investment.  This  does  not  irclude  the  cost  of  other  software  documentation  such 
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as  requirements  specifications  and  design  documents;  those  are  already  included  in  the 
earlier  estimates. 

2 .  Recurring  Costs  (2.0) 

a.  Program  Management  (2.1) 

(1)  Government  (2.1.1).  This  is  the  projected  expense  for  program 
management  of  CAMS.  This  cost  is  estimated  to  be  10  percent  of  the  staff  supporting 
CAMS  in  development,  maintenance,  and  user  support  at  Gunter  SSC. 

b.  Computer  Operations  (2.2) 

(1)  RPC  Operations  (2.2.1,  2.2.2,  and  2.2.3).  Using  data  from  both  the 
Air  Force  Business  Case  Analysis  and  the  RPC  cost  projections,  costs  of  the  SBLC 
Regionalization  were  placed  in  one  of  three  categories:  RPC  CONUS  Operations,  RPC 
USAFE  Operations,  or  RPC  PACAF  Operations.  Overall  personnel  staffing  data  and 
hardwane/software  maintenance  costs  for  each  RPC  were  provided  by  the  Gunter  SCC. 

(2)  Base  Communications  Operations  (2.2.4,  2.2.5,  and  2.2.6).  Costs 
for  personnel  and  equipment  support  at  the  base  level  also  were  placed  in  each  of  the 
categories  CONUS,  USAFE,  or  PACAF.  The  cost  of  the  support  at  each  base  consists  of 
personnel  and  equipment  maintenance  for  communications  and  terminal  support.  The 
personnel  staffing  data  were  provided  by  the  Gunter  SCC,  and  data  from  the  ACC 
provided  average  base  communications  and  terminal  support  costs. 

CAMS’s  share  of  the  RPC  operations  and  base  communication  operations  costs 
was  determined  by  using  the  average  CAMS  workload  as  a  percentage  of  the  total 
workload  in  each  of  the  three  RPC  organizations.  The  data  are  summarized  in  Table  Vn-3. 
(The  workload  is  the  percentage  utilization  of  the  computer  system.) 


Table  VII-3.  CAMS  Workload  In  RPC  Organizations 


CAMS  Utilization 

CPU  Utilization 

CAMS  WoiWoad 

CONUS 

14.1% 

67% 

21.0% 

USAFE 

12.5% 

60% 

20.8% 

PACAF 

13.2% 

71% 

18.6% 
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c.  User  Support  (2.3) 

(1)  Field  Help  Group  (2.3.1).  The  current  Field  Support  Team  at  Gunter 
SSC  takes  problem  telephone  calls  from  data  base  managers  and  offers  help.  We  projected 
that  this  would  continue  at  1 1  staff-years  per  year  at  a  cost  of  $55,660  per  staff-year. 

(2)  Base  Representatives  (2.3.2).  CAMS  data  base  managers  are  located  at 
each  base  to  help  the  users  and  to  serve  as  a  liaison  for  problems.  Historically,  an  average 
of  3.5  data  base  managers  have  been  assigned  to  CAMS  at  each  base.  We  projected  this 
would  continue  at  each  of  89  bases  at  a  cost  of  $55,660  per  year  per  representative. 
Chapter  V  provides  additional  information  about  the  CAMS  base  representatives  and  data 
base  managers. 

(3)  Data  Base  Maintenance  (2.3.3),  This  group  is  located  at  Gunter  SSC  and 
is  charged  with  supporting  the  CAMS  users  and  base-level  data  base  managers.  Key 
responsibilities  include  major  data  base  repair  actions,  performing  system  recoveries, 
investigating  and  resolving  missing  data  transmissions,  supponing  CAMS  data  base 
managers  course  development,  and  performing  detailed  systems  analysis.  Six  people  are 
projected  to  provide  this  service  at  $55,660  per  person  per  year. 

d.  Communications  (2.4) 

(1)  RPC  High-Speed  Links  (2.4.1).  The  RPCs  use  AFNET  communication 
lines  to  transmit  and  receive  data  from  each  of  the  66  CONUS  bases.  The  cost  for  these 
communication  lines  grows  as  more  bases  move  into  the  RPC  operation.  AFNET  is  an  Air 
Force  set  of  communication  facilities  contracted  for  with  various  communication  line 
vendors  and,  in  turn,  leased  or  rented  to  various  Air  Force  organizations.  The 
communication  costs  projected  here  were  based  on  the  same  projections  in  the  Air  Force 
Business  Case  Analysis  for  DMRD  924,  Since  CAMS  accounts  for  about  44  percent  of  all 
the  transactions  between  the  RPC  and  the  base,  the  CAMS  share  of  the  cost  was  allocated 
at  44  percent  of  the  cost. 

(2)  High-Speed  REMIS  Link  (2.4.2).  The  study  team  is  projecting  the  cost 
of  a  high-speed  (56  Kbps)  line  from  each  RPC  to  REMIS  to  provide  the  high-speed 
turnaround  needed  for  the  enhanced  data-editing  function  previously  outlined.  A  monthly 
rate  of  $3,500  for  each  RPC  was  assumed. 

e.  Software  Maintenance  (2.5) 

(1)  Applications  Software  (2.5.1).  This  cost  element  provides  funds  for  the 
maintenance  of  CAMS  software.  Software  maintenance  includes  correcting  and  testing 
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software  errors  reported  by  users  (Discrepancy  Reports  or  DIREPs),  making  modifications 
to  the  CAMS  data  base  to  enhance  performance,  or  add/correct  data  relationships.  Recent 
CAMS  history  indicates  that  about  40  percent  of  the  CAMS  staff  (45  staff  members)  are 
needed  to  address  the  current  maintenance  workload.  The  study  team  estimates  that  this 
level  of  effort  will  be  required  for  the  foreseeable  future.  Since  CAMS  consists  of 
approximately  1.1  million  lines  of  unique  source  code,  this  results  in  the  maintenance  of 
24,961  unique  lines  of  source  code  per  staff  year.  This  is  in  line  with  typical  industry 
averages  for  data  base  applications,  which  vary  between  20,(X)0  and  32,000  unique  lines  of 
source  code  per  staff  year.*  This  work  will  be  performed  by  a  mixture  of  military  and 
civilian  personnel  at  an  average  rate  of  $55,660  per  year. 

(2)  System  Integration  and  Test  (2.5.2).  This  function  is  performed  by  the 
QT&E  group  at  the  Gunter  SSC  (about  75  persons  in  total).  The  cost  projections  were 
based  on  a  prorated  share  of  the  current  QT&E  support  We  estimated  nine  of  the  QT&E 
personnel  would  be  required  for  maintenance  support  at  a  cost  of  $55,660  per  staff-year. 

(3)  Data  Base  Management  (2.5.3).  This  effort  is  staffed  by  the  CAMS 
organization  at  the  Gunter  SSC  and  includes  making  modifications  to  the  CAMS  data  base 
to  enhance  performance,  or  add/correct  data  relationships  by  making  data  base  schema 
modifications.  The  study  team  estimated  this  effort  to  require  four  experienced  data  base 
analysts  at  an  avenge  rate  of  $55,660  per  year. 

(4)  Documentation  (2.5.4).  These  funds  include  resources  to  modify  and 
update  the  software  documentation  in  accordance  with  the  changes  due  to  the  software  and 
data  base  maintenance.  This  was  estimated  to  require  five  staff-years  per  year. 

f.  Training  and  Travel  (2.6) 

Training  and  travel  costs  of  the  CAMS  users  are  incurred  by  the  individual  bases 
and  were  not  included  in  our  estimate.  The  CAMS  base  representatives  will  require  training 
on  a  continuous  basis  as  personnel  are  reassigned.  The  training  and  travel  costs  are 
estimated  to  be  2  percent  of  the  funds  under  the  control  of  CAMS  management 


*  *  Bodun.  B.W.,  Software  Engineering  Economics,  Prtntice-Hall,  1981,  p.  541. 
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D.  COST  ESTIMATE  FOR  REMIS 


1 .  Assumptions 

The  cost  analysis  for  REMIS  is  summariz/*d  in  Table  VII-4.  The  iiumbers  shown  in 
parentheses  in  the  subsections  that  follow  correspond  to  the  numbers  to  the  left  of  the  cost 
categories  in  the  table.  The  cost  analysis  assumed  that  REMIS  will  continue  to  be 
maintained  by  Litton  Computer  Services  (LCS)  at  their  facility  in  Dayton,  Ohio.  IDA  added 
investment  dollars  to  address  two  major  problems  with  the  system.  One  problem  was  slow 
performance  in  generating  reports.  We  assumed  that  performance  must  be  addressed  via 
additional  hardware  and  via  software  rewrites.  A  second  area  of  improvement  is  to  expand 
the  scope  of  data  that  can  be  accessed  through  REMIST ALK.  The  cost  to  address  both  of 
these  issues  is  included  in  the  cost  element  dealing  with  nonrecurring  software  costs  (1.2). 

REMIS  was  originally  projected  to  reach  MAISRC  Milestone  HI  approval  in 
September  1993.  According  to  the  REMIS  Program  Director,  the  date  for  Milestone  HI 
approval  is  now  projected  for  October  1994,  with  fuD  operational  capability  (FOC)  planned  ' 
for  January  1995.  The  cost  estimates  reflect  this  latest  schedule. 

2.  Sources  of  Information  for  REMIS  Cost  Estimates 

The  IDA  team  consulted  several  sources  of  information  in  preparing  the  cost 
estimates  for  REMIS.  Detailed  information  was  available  from  the  Program  Office  Estimate 
(POE)  and  Independent  Cost  Estimate  (ICE)  prepared  for  the  MAISRC  MUestone  m.  More 
recent  cost  projections  were  obtained  from  LCS.  Discussions  by  telephone  and  in  person 
were  held  with  members  of  the  REMIS  Program  Office  as  well  as  with  Litton  personnel. 
Our  general  approach  was  to  take  data  from  tl^se  sources  as  inputs  to  the  cost  analysis.  For 
mc^t  cost  categories,  we  estimated  the  costs  independently  as  well. 

3 .  Nonrecurring  Costs 

The  nonrecurring  costs  are  divided  into  hardware,  software,  communications, 
training  support  for  OT&E,  and  security.  They  are  discussed  in  that  order. 

a.  Hardware  (1.1) 

(1)  Mainframes  (1.1.1).  In  order  to  adequately  address  REMIS’s  performance 
problems,  it  is  IDA’s  judgment  that  another  Tandem  mainframe  will  be  needed.  We 
assumed  a  12-processor  machine  equivalent  to  HQl  for  $5  million.  This  estimate  includes 
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the  raainframe,  disk  storage,  control  units,  and  a  communications  processor.  A  $1.31 
million  upgrade  is  included  in  the  estimate  after  seven  years  (assuming  a  20-percent 
compound  annual  cost  reduction). 

We  are  also  assuming  there  will  be  an  upgrade  of  all  other  Tandem  mainframes  in 
FY  1996  at  a  total  cost  of  $6  million. 

b.  Software  (1.2) 

(1)  Applications  Software  (1.2.1).  This  category  contains  the  estimated  cost 
for  additional  work  that  in  IDA’s  judgment  is  necessary  to  bring  REMIS  up  to  an 
acceptable  level  of  operational  performance. 

Performance  Improvement.  This  cost  element  covers  work  needed  to  reduce 
the  overhead  associated  with  system -to-system  input  (SSI)  and  the  network  control  system 
(operations  overhead)  so  that  additional  resources  will  be  available  for  report-generation. 
This  will  entail  a  great  deal  of  performance  measurement  and  analysis  and  some  software 
re-design  and  re-coding.  We  estimated  15  staff-years  for  each  of  three  years  at  $132,480 
per  staff-year  resulting  in  an  total  estimated  cost  of  $1,987,200  per  year. 

REMISTALK  Improvement,  This  cost  element  encompasses  work  to  expand 
the  scope  of  REMISTALK  and  reduce  the  time  required  to  access  data.  The  problems  and 
improvements  needed  are  discussed  at  length  in  Chapter  V.  We  estimated  10  people  for 
three  years  at  a  cost  of  $1,324,8{X)  per  year. 

(2)  Data  Base  Initialization  (1.2.2).  Once  REMIS  reaches  FOC,  the 
GCSAS  subsystem  will  contain  configuration  data  on  all  Air  Force  weapon  systems.  Effort 
will  have  to  be  expended  to  load  the  appropriate  configuration  tables  into  the  data  base. 
According  to  Litton  and  the  REMIS  PMO,  this  will  be  done  by  existing  Air  Force 
personnel  (equipment  specialists)  residing  at  the  Air  Logistics  Centers  (ALCs)  for  existing 
weapons  systems.  For  new  weapon  systems,  the  tables  will  be  loaded  by  personnel  in  the 
SPOs  and  by  contractors.  As  far  as  we  could  determine,  no  funding  has  been  set  aside  for 
this  task,  but  it  is  work  that  will  have  to  be  done. 

Even  though  Air  Force  personnel  will  be  loading  the  configuration  tables,  we 
expect  it  will  be  necessary  for  personnel  from  Litton  to  help  ensure  that  the  information  is 
gathered  and  entered  accurately.  This  will  require  skilled  engineering  persoimel  who 
understand  the  configuration  issues  related  to  specific  weapon  systems  and  their 
implications  for  the  REMIS  data  base.  Because  this  information  is  critical  and  has  proved  to 
be  difHcult  to  gather  in  the  past,  we  assume  a  total  of  15  staff-years  during  FY  1994  and  10 
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staff-years  during  FY  1995  to  assist  in  gathering  this  information.  The  estimated  costs  per 
year  are  as  follows; 

FY  1994:  15  staff  years  at  $132,480  per  year  =  $1,987,200  and 
FY  1995:  10  staff  years  at  $132,480  per  year  =  $1,324,800. 

(3)  Documentation  (1.2.3).  This  category  covers  the  cost  of  producing  and 
distributing  user’s  manuals.  We  estimated  costs  at  10  percent  of  the  application  software 
investment.  This  did  not  include  the  cost  of  other  software  documentation  such  as 
requirements  specifications  and  design  documents;  those  were  included  in  the  previous 
estimates. 

c.  Communications  (1.3) 

The  costs  of  leasing  communications  lines  from  the  ALCs  to  Litton’s  facility  in 
Dayton  is  included  under  recurring  costs. 

d.  User  Training  (1.4) 

According  to  Litton,  there  are  no  funds  budgeted  for  user  training.  Litton  estimated 
that  there  are  approximately  1,100  users  to  be  trained  on  GCSAS.  We  estimated  $250, 0(X) 
during  FY  1995  and  FY  1996  for  preparation  and  delivery  of  training  at  user  sites. 

e.  Support  for  OT&E  (1.5) 

GCSAS  is  currently  undergoing  an  operational  test  and  evaluation  (OT&E)  in 
preparation  for  the  MAISRC  Milestone  ID.  This  cost  category  covers  the  effort  expended 
by  the  PMO  and  Litton  in  support  of  that  effort.  We  estimate  the  cost  of  that  effort  at  $1.4 
million. 

We  have  added  an  additional  $1.25  mUlion  in  FY  1997  for  the  REMIS  PMO  and 
Litton  support  of  a  combined  CAMS/REMIS  OT&E.  We  assumed  that  the  CAMS/REMIS 
OT&E  will  not  begin  until  the  needed  improvements  are  made  to  both  CAMS  and  REMIS. 

4.  Recurring  Costs  (2.0) 

The  recurring  costs  are  divided  into  program  management,  computer  operations, 
user  support,  communications,  software  maintenance,  training  and  travel,  and  continuity  of 
service.  They  are  discussed  in  that  order. 
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a.  Program  Management  (2.1) 

(1)  Government  (2.2.1).  Costs  for  the  government  program  management  are 
estimated  at  10  percent  of  total  contractor  labor  (including  contractor  program 
management). 

(2)  Contractor  (2.2.1)  Costs  for  contractor  program  management  are  estimated 
at  5  percent  of  total  contractor  labor. 

b.  Computer  Operations  (2.2) 

(1)  Central  Computer  (2.2.1).  The  IDA  team  estimated  that  a  total  of  19 
personnel  will  be  required  over  three  shifts  to  operate  the  Tandem  computers  located  in 
Dayton.  Assuming  an  average  burdened  staff-year  of  $77,280,  the  total  is  $1,468,320  per 
year. 

Hardware,  Software,  and  Utilities  covers  the  cost  of  maintenance  for  the 
processors  at  Litton’s  facility  in  Dayton.  It  includes  $320,000  per  year  for  maintenance  of 
HQl  and  HQ2  plus  an  additional  $220,0(X)  per  year  for  maintenance  of  the  additional 
mainframe  (HQl  equivalent).  This  category  also  includes  an  estimate  of  $91,000  per  year 
for  system  software  and  utilities. 

(2)  ALCs  (2.2.2).  The  Tandem  processors  at  the  Air  Logistics  Centers  are 
operated  in  a  “lights  out”  mode  from  the  Network  Control  Center  (NCC)  in  Dayton.  For 
that  reason,  no  costs  were  included  in  the  operational  staff  category. 

Maintenance  of  the  13  Tandem  processors  operating  at  the  ALCs  was  estimated  at 
$235,000  per  year.  We  also  added  $39,000  for  system  software  and  utilities. 

c.  User  Support  (2.3) 

User  support  takes  the  form  of  phone  operators  at  the  Network  Control  Center 
(NCC)  in  Dayton,  Ohio,  plus  technical  analysts  to  help  resolve  user  problems.  The  analysis 
assumes  a  total  of  six  phone  operators  providing  24-hour-a-day  coverage  at  a  burdened  rate 
of  $30  per  hour  plus  15  percent  for  G&A  plus  fee.  That  totals  $34.50  or  $66,240  per 
operator  per  year.  The  analysis  also  assumes  five  technical  analysts  at  a  burdened  rate  of 
$132,480  per  year.  The  total  cost  of  user  support  is  $1,059,840  per  year. 

d.  Communications  (2.4) 

This  category  covers  the  cost  of  leasing  communications  lines  between  each  of  the 
ALCs  and  the  LCS  facility  in  Dayton.  Based  on  information  provided  by  LCS,  the 
cmnmunications  charges  are  estimated  at  $155,000  per  year. 
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e.  Software  Maintenance  (2.5) 

(1)  Applications  Software  (2.5.1).  The  cost  of  maintaining  the  application 
software  was  estimated  assuming  that  25,000  lines  of  code  can  be  maintained  per  staff- 
year.  According  to  LCS,  REMIS  contains  736,000  lines  of  COBOL  code  (including 
declarations  and  excluding  comments).  Dividing  736,000  by  25,000  gives  29.44  staff- 
years,  which  was  rounded  up  to  30.  Assuming  30  people  at  $132,480  per  year  gives  an 
annual  cost  of  $  3,974,400. 

(2)  Data  Base  Management  (2.5.2).  The  information  associated  with  the 
complex  weapon  systems  supported  by  REMIS  is  constantly  evolving  as  the  approved  and 
actual  configuration  changes,  as  technical  orders  are  issued,  and  so  on.  This  category 
covers  the  resources  required  to  keep  the  data  base  current  as  well  as  normal  data  base 
maintenance.  We  estimated  10  data  base  analysts  at  $132,480  per  year  for  a  total  of 
$1,324,800. 

(3)  Documentation  (2.5.3).  The  recurring  documentation  costs  represent 
changes  that  must  be  made  to  documentation  whenever  modifications  are  made  to  REMIS 
that  affect  the  users.  These  could  be  modifications  as  a  result  of  resolving  a  problem  report 
or  modifications  that  represent  functional  enhancements.  We  estimated  them  at  5  percent  of 
the  cost  of  application  software  maintenance. 

f.  Training  and  Travel  (2.6) 

This  category  covers  the  cost  of  on-going  user  training  plus  general  travel.  These 
costs  have  been  estimated  at  10  percent  of  the  cost  of  application  software  maintenance. 

g.  Continuity  of  Service  (2.7) 

In  the  event  of  a  catastrophe  (e.g.,  fire  or  tornado)  in  Dayton  that  destroyed  the 
computer  facilities,  LCS  plans  to  switch  processing  to  the  Oklahoma  City  ALC,  which  has 
five  processors  compared  to  the  other  four  ALCs  with  two  processors  each.  REMIS  would 
operate  there  in  a  degraded  mode  until  additional  processors  could  be  obtained.  (By  way  of 
comparison,  REMIS  operates  on  a  total  of  20  processors  on  HQl  and  HQ2.)  Because 
REMIS  is  not  critical  for  day-to-day  base-level  flight  operations,  we  viewed  this  as  an 
acceptable  backup  plan  and  added  no  costs  to  this  category. 

h.  Security 

A  Tandem  product  called  SAFEGUARD  will  enable  REMIS  to  meet  Category  2 
security  requirements.  The  cost  for  this  product  for  HQl,  HQ2,  and  the  processors  at  the 
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ALCs  is  $28,284  per  year.  Because  we  do  not  have  comparable  information  for  aU  systems 
and  because  the  cost  is  so  small,  we  deliberately  excluded  security  costs  from  Table  Vll-4. 

E.  COST  ESTIMATE  FOR  TICARRS 

1 .  Assumptions 

Our  cost  analysis  was  based  on  the  assumption  that  TICARRS  will  continue  to  be 
maintained  by  Dynamics  Research  Corporation  (DRC)  in  Andover,  Massachusetts.  This 
need  not  be  the  case.  TICARRS  could  be  taken  in-house  by  the  Air  Force  or  operated  by 
another  contractor,  but  it  would  be  difficult  to  assess  the  risk  or  estimate  the  costs  of 
expanding  TICARRS  under  such  circumstances. 

We  assumed  that  all  base-level  input  will  be  direct  to  TICARRS  rather  than  through 
CAMS.  The  base- level  daily  transaction  rate  will  be  less  than  the  FY  1993  averages  for  the 
next  ten  years  due  to  base  closures  and  unit  de-activations. 

DRC  will  provide  domestic  communications;  the  Air  Force  will  provide  overseas 
communications.  The  domestic  communications  prices  were  based  on  Sprint  rates  in  use  by 
DRC.  It  is  feasible  to  use  the  AFNET  instead;  however,  we  did  not  evaluate  the  cost  of  the 
AFNET  services.  (Based  on  the  AFNET  costs  for  the  RPCs,  it  is  likely  that  the  monthly 
rates  would  decrease  compared  to  the  Sprint  charges).  Data  Communication  Processors 
(DCP),  terminals,  and  printers  currently  at  the  bases  would  be  used,  and  the  staff  now  in 
the  Air  Force  plan  would  support  the  equipment  TICARRS  will  be  expanded  to  include  all 
Air  Force  weapon  systems  except  strategic  airlift  aircraft  (which  are  covered  by  CAMS). 
This  expansion  will  include  a  total  of  44  types  of  aircraft,  missiles,  and  1,700 
communication-electronics  end  items.  A  total  of  254  units  (squadrons)  will  change  over 
from  CAMS  to  TICARRS.  The  completed  cost  element  structure  is  shown  in  Table  VII-5. 
As  before,  the  numbers  to  the  left  of  the  cost  categories  are  noted  in  parentheses  in  the 
discussion  of  cost  that  follows. 

In  order  to  expand  TICARRS  to  cover  all  weapon  systems  now  covered  by  CAMS 
and  89  Air  Force  bases,  there  are  several  major  steps  that  must  be  taken: 

•  Additional  hardware  must  be  purchased  in  order  to  handle  the  greatly  increased 
demands  on  processing  and  data  storage. 
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Table  VII-5.  Cost  Estimate  for  TICARRS 

_ Cost  in  Millions  of  1994  Dollars 
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2.2  Computer  C^)eiatk>ns 
2.2.1  Central  Computer 

Computer  Usage  0.50  0.50  1  00 

Opci^onalStaff  1.68  1.68  1.95  2.55  2.55  2.55  2.55  2.55  2.55  2.55  23.17 

Hardware,  Software,  and  Utilities  0.72  0.72  0.72  1.45  1.45  1.45  1.45  1.45  1.45  1.57  12.43 
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•  Enhancements  will  be  required  to  the  application  software.  As  discussed  in 
Chapters  V  and  VI,  eight  functional  enhancements  have  been  identified. 

•  The  data  base  must  be  initialized  for  each  of  the  weapon  systems  and 
associated  trainers  and  simulators,  and  for  communications-electronics. 

•  Data  must  be  loaded  from  REMIS. 

•  Additional  communications  must  be  added. 

•  A  total  of  254  units  must  be  activated  and  users  trained. 

The  effects  of  these  steps  on  cost  are  discussed  in  the  subsections  on  nonrecurring 
and  recurring  costs  that  follow. 

In  estimating  the  nonrecurring  costs,  we  assumed  the  schedule  that  was  discussed 
in  Chapter  VI.  Specifically,  we  assume  that,  as  an  Air  Force  standard  system,  TICARRS 
would  be  subject  to  an  Operational  Test  and  Evaluation  (OT&E)  and  to  MAISRC  approval. 
The  cost  analysis  for  expanding  TICARRS  assumes  that  a  contract  to  perform  the  work 
would  be  awarded  by  1  November  1994.  Eight  months  would  be  spent  in  enhancing  the 
functionality  of  TICARRS  to  take  over  the  functions  of  CAMS/REMIS.  In  addition,  in 
preparation  for  the  OT&E,  the  data  base  would  be  initialized  for  two  weapon  systems  and 
CAMS  data  from  one  base  would  be  loaded  onto  TICARRS,  This  would  be  accomplished 
by  1  October  1995.  The  OT&E  would  then  occur  during  October  through  December  at  the 
one  base  for  the  two  weapon  systems.  Data  base  initialization  and  data  loading  would  occur 
throughout  the  Air  Force  during  FY  1996  and  FY  1997. 

There  are  two  areas  where  delays  in  the  assumed  schedule  may  occur:  delays  in  the 
acquisition  process  and  delays  in  the  nonrecurring  development  program.  We  have 
included  a  sensitivity  analysis  to  address  this  issue. 

2.  Nonrecurring  Costs  (1.0) 

a.  Hardware  (1.1) 

For  TICARRS  to  provide  the  same  or  better  service  to  all  Air  Force  users  as 
CAMS/REMIS  does,  hardware  must  be  added.  The  current  TICARRS  hardware  is  as 
follows: 

•  Mainframe:  Bull  9094  DPS^  (36-MIP  four-way  processor,  192-megabyte 
RAM). 


2  This  otainframe  ctxnputer  is  not  used  full-time  to  support  tte  TICARRS  workload. 
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•  Cost:  $3.0  million  (1993  cost  including  mainframe  and  disk  storage). 

•  Communications  Processors:  two  Unisys  DCP-30  processors. 

•  Communication  Lines:  57  lines  at  9.6  kilobyte  (leased  from  Sprint). 

(1)  Mainframes  (1.1.1).  Computer  power  must  be  about  four  times  the 
existing  level.  Allowing  for  a  20-percent  compound  cost-reduction  rate  through  1996,  this 
requires  an  investment  or  nonrecurring  cost  of  $5.54  million.  An  additional  $1.1  million  in 
2003  is  needed  for  replacement  cost. 

Historically,  the  daily  transaction  rale  for  CAMS  has  been  1.1  million  transactions 
per  day  over  an  8-hour  period  (first  shift).  An  estimated  1.6  million  transactions  per  day  are 
processed  over  a  24-hour  period  (based  on  data  collected  by  USAF  SSC  over  the  period 
January  through  February  1993  (8  weeks).  The  daily  transaction  rate  during  first  shift  is 
expected  to  drop  from  1.1  million  to  1.0  million  due  to  the  number  of  base  closings 
planned  through  1994.  The  projected  transaction  rate  is  based  on  the  number  of  bases 
dropping  from  107  to  89. 

Data  collected  by  DRC  during  the  TICARRS  92  assessment  at  Seymour  Johnson 
AFB  measured  the  central  processor  utilization  rates  for  TICARRS  to  be: 

•  2.54  percent  (average)  over  a  24-hour  period  to  process  an  average  40,000 
transactions, 

•  4.60  percent  (average)  over  an  8-hour  period  (peak)  to  process  an  average  of 
20,000  transactions,  and 

•  99  percent  of  TICARRS  transactions  had  response  times  of  less  than  3.24 
seconds.  Assume,  therefoie,  that  the  central  processor  is  used  no  more  than  an 
average  of  65  percent  during  peak  shift  in  order  to  achieve  a  3-second  response 
time.  Keeping  the  average  utilization  at  65  percent  permits  the  computer  to 
operate  at  higher  utilization  rates  for  brief  periods  to  work  off  messages  in 
queues  and  surges  in  transaction  traffic. 

The  ratio  of  average  peak  utilization  to  average  peak  transactions  is  used  to  calculate 
the  additional  computer  use  required  for  1.0  million  transactions  processed  by  all  the 
CAMS  installations.  Because  the  Seymour  Johnson  data  indicate  that  the  average  number 
of  TICARRS  transactions  exceed  the  average  number  of  CAMS  transactions  by  5.6 
percent,  we  increased  the  total  number  of  transactions  by  5.6  percent  to  compensate: 

1.056  million  CAMS  transactions  (average  peak)  •<-  20,000  average 
transactions  (peak)  =  52.8; 

52.8  X  4.6  percent  a  243  percent  utilization. 


vn-22 


Total  CAMS  support  requires  243  percent  utilization,  or  87.4  million  instructions 
per  second  (MIPS),  to  support  1.056  million  transactions  during  peak  or  first  shift. 

To  provide  the  computing  power  to  support  REMIS  users,  the  computer  utilization 
for  those  REMIS  activities  not  included  above  were  added: 

•  2.5  percent  utilization  for  file  maintenance,  table  maintenance  queries,  error 
correction  and  electronic  mail; 

•  2.5  percent  utilization  for  processing  and  transmitting  data  outbound  to  other 
systems;  and 

•  10  percent  utilization  for  user  reports. 

Thus,  the  total  additional  REMIS  support  is  15  percent. 

Total  TICARRS  power  needed  to  support  CAMS/REMIS  users  is  258  percent  (243 
percent  for  CAMS  plus  15  percent  for  REMIS),  and  the  maximum  average  computer 
utilization  during  peak  hours  is  65  percent 

The  increase  needed  in  TICARRS’s  computer  system  is:  258  percent  +  65  percent  = 
3.96.  Measured  in  terms  of  existing  computers,  TICARRS  will  need  3.96  x  36  MIPS  = 
142.8  MIPS. 

The  cost  of  the  current  computer  is  $3.0  million;  however,  the  1996  price  of  the 
same  computer  is  projected  to  be  $1,536  million  (20  percent  compound  reduction  rate)  and 
$1,228  million  in  1997,  The  cost  of  the  computer  system  was  spread  over  two  years  to 
match  the  growing  workload  caused  by  adding  bases  to  the  TICARRS  system  during  1996 
and  1997,  resulting  in  a  cost  of  $5.54  million. 

b.  Software  (1.2) 

(1)  Application  Software  (1.2.1). 

Functional  Enhancements.  A  total  of  eight  functional  enhancements  have  been 
identified  for  TICARRS: 

•  CEMS  interface. 

•  SBSS  interface, 

•  personnel  and  training, 

•  aerospace  ground  equipment  (AGE). 

•  production  management  (comparable  to  CAMS  380  screen), 

•  maintenance  snapshot  (comparable  to  CAMS  122  screen). 
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automated  aircraft  forms,  and 

Product  Quality  Deficiency  Reports  (PQDRs). 


Before  discussing  the  estimates  for  including  these  functions  in  TICARRS,  it  is 
necessary  to  discuss  lines-of-code  counts  because  they  are  a  major  cost  driver  in  the 
implementation  of  additional  functionality.  There  is  a  large  discrepancy  between  the  lines- 
of-code  count  in  TICARRS  and  the  combined  count  for  CAMS/REMIS;  this  discrepancy 
has  been  the  impetus  for  a  great  deal  of  debate.  The  Air  Force  and  LCS  make  the  argument 
that  the  number  of  lines  of  code  required  to  implement  the  above  functions  in  CAMS  or 
REMIS  will  be  the  number  required  for  TICARRS.  We  considered  these  arguments  but 
rejected  them  because  the  designs  of  the  systems  are  vastly  different. 

CAMS  contains  1.1  million  unique  lines  of  COBOL  code,  REMIS  contains  736 
thousand,  and  TICARRS  contains  417  thousand.  On  the  surface,  it  appears  that  TICARRS 
contains  less  than  25  percent  as  much  code  as  CAMS/REMIS.  Central  to  the  design  of 
TICARRS  is  a  tool  called  Middleware,  which  provides  sets  of  reusable  code  and  program 
“shells”  to  handle  the  common  data  base  functions  of  displaying  information  to  the  user  on 
the  screen,  updating  information  in  the  data  base,  leuieving  information  from  the  data  base, 
and  processing  user  input.  The  use  of  Middleware  allows  much  of  the  functionality  of 
TICARRS  to  be  table-driven.  One  line  of  a  table  can  replace  many  lines  of  COBOL  code. 
With  Middleware,  the  programmers  write  only  the  application-specific  code  for  any  given 
function.  It  is  important  to  note  that  the  total  COBOL  for  TICARRS  (i.e.,  the  COBOL 
written  by  the  programmers  plus  the  COBOL  generated  by  Middleware)  is  actually  more 
voluminous  than  CAMS/REMIS  (2.44  million  lines  of  COBOL  versus  1.84  million  for 
CAMS/REMIS).  However,  it  is  the  programmer-generated  code  and  not  the  total  COBOL 
that  the  programmers  have  to  write  and  maintain,  and  that  is  the  lines-of-code  count  we 
considered  for  estimating  the  effort  involved  in  the  functional  enhancements. 

As  an  illustration  of  the  differences  in  lines  of  code  between  the  systems,  we 
gathered  lines-of-code  counts  for  functions  implemented  in  TICARRS  and  in 
CAMS/REMIS.  Table  VII-6  shows  these  counts  for  four  such  functions. 

For  the  above  functions  overall,  there  is  an  approximately  3:1  ratio  between  the 
count  of  programmer-written  lines  for  CAMS/REMIS  versus  TICARRS.  Note  that  the  total 
COBOL  count  for  HCARRS  surpasses  CAMS/REMIS. 

In  estimating  the  resources  required  for  implementing  each  of  the  eight  functional 
enhancements,  we  looked  at  the  design  of  TICARRS  and  at  how  the  enhancements  would 
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be  implemented  within  that  design.  That  is  different  from  how  they  are  or  might  be 
implemented  within  another  system. 


Table  Vil-6.  Comparison  of  Lines-of-Code  Counts 


CAMS 

REMIS 

TICARRS 

Function 

COBOL 
Source  Lines 

COBOL 
Source  Lines 

Programmer- 
Generated  Lines 

COBOL 
Source  Lines 

Auto  Debriefing 

55,280 

16,322 

51,339 

Equipment  Transfer 

18.682 

6,617 

19,473 

Utilization  Reporting 

28.737 

5.911 

26.762 

InspecUon/Time  Change 

5.414 

6.928 

28,634 

Total 

Total  CAMS/REMIS 

73.962 

108.113 

34,151 

35,778 

126,208 

Source:  The  iines-of-code  counts  for  CAMS  and  REMIS  were  taken  from  “Impact  Information 
Regarding  FY  93  HAC  Report  Language  on  CAMS/REMIS  and  TICARRS"  by  Robbins-Gir 
Incorporated.  1992. 


As  one  input  to  our  estimate,  we  obtained  a  formal  impact  analysis  for  each 
enhancement  from  DRC.  Each  impact  analysis  included  a  detailed  description  of  the 
function  to  be  added,  a  description  of  the  programs  or  modules  within  TICARRS  that  are 
impacted,  an  estimate  of  the  number  of  lines  of  code  to  be  added  or  changed  within  each 
module,  the  layout  of  all  new  and  modified  screens,  a  description  of  any  changes  to  the 
data  base,  and  an  estimate  of  the  staff  hours  required  to  specify,  design,  implement, 
document,  and  test  the  enhancement. 

Our  first  step  was  to  evaluate  the  credibility  and  reliability  of  the  impact  analyses  as 
a  basis  for  estimating.  The  impact  analyses  were  reviewed  by  functiona]  analysts  from  IDA 
to  assess  their  validity  in  terms  of  adequately  describing  the  function  to  be  implemented. 
Secondly,  we  evaluated  DRC’s  record  in  using  the  impact  analyses  as  a  basis  for  resource 
estimates.  We  obtained  historical  data  from  DRC,  which  allowed  us  to  compare  their 
estimates,  based  on  similar  types  of  impact  analyses,  with  actual  hours  charged  to  the  Air 
Force.  Over  a  total  of  64  efforts,  which  included  a  mix  of  functional  enhancements, 
performance  enhancements,  and  defects  corrections,  18  of  the  64  had  been  under-estimated 
by  DRC  and  46  had  been  over-estimated.  On  average,  DRC  completed  the  work  with  13 
percent  fewer  staff  hours  than  estimated. 

Having  established  that  the  impact  analyses  were  adequate  from  a  functional 
standpoint  and  that  they  served  as  a  basis  for  accurate  resource  estimates,  we  used  them  to 
derive  lines-of-code  estimates  for  entry  into  a  software  cost  model,  SPQR.  This  model  was 
chosen  because  it  provides  a  straightforward  means  of  modeling  the  enhancements  as  they 
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would  be  implemented  in  TICARRS.  In  deriving  the  estimates,  we  entered  three  different 
lines-of-code  counts; 

•  the  size  of  the  base  code  involved  in  each  of  the  enhancements  (The  base  code 
encompasses  any  and  all  modules  for  which  any  lines  are  added  or  modified. 
That  code  is  relevant  because  it  must  be  handled  and  tested,  thereby  adding  to 
the  workload.), 

•  the  size  of  reused  code,  in  this  case  a  count  of  the  lines  of  the  Middleware 
shells,  and 

•  the  size  of  the  new  or  changed  lines. 

Table  VII-?  shows  these  three  counts  for  each  enhancement.  The  table  also  shows 
the  number  of  staff  hours  estimated  by  the  model  for  each  enhancement.  The  total  costs  for 
the  functional  enhancements  were  estimated  by  multiplying  the  number  of  staff  hours 
estimated  by  SPQR  (23,917)  by  $69  per  hour  for  a  total  of  $1,650,273.  An  additional  10 
percent  was  added  for  generating  or  modifying  the  Middleware  tables,  for  a  total  of 
$1,815,300. 


Table  Vll-T.  Estimated  Lines  of  Code  and  Staff  Hours 
Required  for  Functional  Enhancements 


Functional  Enhancement 

Base  Code 
Size 

Reused  Code 
Size 

New/Changed 
Code  Size 

Estimmed 
Staff  Hours 

Production  Management 

117,755 

14,781 

4,215 

2,496 

CEMS  Interface 

75,440 

7,669 

5,540 

3,338 

SBSS  Interface 

135,804 

19,329 

12,860 

8,322 

Personnel/Training 

0 

6,462 

7,950 

4,740 

Maintenance  Snapshot 

3,318 

681 

595 

320 

Automated  Aircraft  Forms 

39,470 

9,358 

5,110 

2,987 

PQDR 

26,119 

4,435 

1,503 

878 

AGE 

25,074 

6,389 

1,450 

830 

Total 

39,223 

23,917 

We  assumed  that  all  eight  enhancements  will  be  carried  out  in  FY  1995.  There  do 
not  appear  to  be  sequential  dependencies  among  them,  so  we  assumed  that  they  can  be 
implemented  in  parallel. 

REMtS  interfaces.  Although  TICARRS  would  replace  a  number  of  existing 
systems  (as  will  REMIS),  thirteen  software  system  interfaces  would  not  be  replaced. 
Programming  effort  will  be  necessary  to  accept  data  from  those  systems  and  to  provicte 
them  with  the  data  they  need,  in  the  format  they  need.  Tl^  thirteen  systems  are  as  follows: 

•  D086A — Individual  Aircraft  Service  Life  Monitoring  System, 
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•  K002 — Peacetime  Programming  Computational  System, 

•  K008 — Aerospace  Vehicle  Percent  Flying  Hour  Program  System  for  AFMC, 

•  G08 1— C  AMS  for  Airlift. 

•  D087F — Weapon  System  Management  Information  System, 

•  D160B — Visibility/Management  of  Operation  Support  Cost  System, 

•  D165B — Aerospace  Vehicle  and  Mission  Capability  System, 

•  D200 — ^Requirements  Data  Bank, 

•  GOOIC — Contractor  and  Depot  Maintenance  Data  Collection  System, 

•  IX)43 — Master  Item  Identification  Control  System, 

•  D038 — Precision  Measurement  Equipment  Calibration  Interval  Analysis, 

•  D075 — Logistics  Management  Data  Bank,  and 

•  GOl  2 — Computational  Support  for  Create  Engineer  Support 

Each  interface  was  estimated  to  require  .75  staff-years  to  develop  at  $132,480  per 
staff  year.  We  assume  that  thirteen  interfaces  would  be  implemented  in  FY  1995. 

Load  REMIS  Data.  This  category  covers  the  cost  of  loading  historical  data  from 
REMIS  into  TICARRS.  We  assume  that  this  will  be  done  electronically  with  some  manual 
effort  required  to  resolve  data-loading  problems.  Programming  effort  must  be  expended  to 
write  code  to  translate  the  REMIS  files  into  a  format  that  can  be  read  by  TICARRS.  We 
estimated  ten  staff-years  at  $132,480  per  staff-year. 

c.  Data  Base  Initialization  (1.2.2) 

The  data  base  must  be  initialized  for  each  new  weapon  system.  The  cost  will  vary 
with  the  complexity  of  the  system  and  with  the  number  of  serially  tracked  items.  The 
following  information  must  be  added  to  the  data  base  for  each  new  weapon  system: 

•  description  of  equipment: 

work  unit  codes  (for  test  stations  as  weU  as  aircraft), 
part  numbers, 
serial  numbers,  and 

hierarchical  relationship  between  WUCs; 

•  approved  configuration; 

•  quantity  of  components  per  weapon  system ; 

•  identify  serially  tracked  items; 
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•  phase  inspections; 

•  open  TCTOs; 

•  special  inspections; 

•  location  of  equipment; 

•  contractors  requiring  access;  and 

•  PQDR  information. 

Estimates  for  each  weapon  system  are  shown  in  Table  VII-8.  These  include  aircraft, 
trainers,  and  simulators.  These  estimates  were  derived  by  estimating  the  staff-hours 
required  for  each  weapon  system,  taking  into  account  the  complexity  of  the  avionics  and 
the  automatic  test  equipment,  the  complexity  of  possible  missions,  and  the  complexity  of 
the  configuration  to  be  tracked.  We  included  additional  costs  for  variations  in  block 
numbers  and  model  designations.  Note  that  the  TICARRS  data  base  has  already  been 
initialized  to  a  large  extent  for  the  F-16,  F-15,  and  F-1 17,  with  the  result  that  the  estimated 
cost  for  these  systems  is  less  than  would  be  expected  for  a  system  that  was  not  in  the  data 
buSc.  We  also  included  an  estimate  for  communicadons-electronics.  We  assumed  data  base 
initialization  would  average  $1,000  for  each  of  the  1,700  types  of  communications- 
electronic  equipment  for  a  total  of  $1,700,000.  The  total  cost  of  data  base  initialization  was 
estimated  to  be  $9,858,000. 

We  assumed  that  data  base  initialization  will  begin  in  the  fourth  quarter  of  FY  1995 
and  continue  through  the  first  quarter  of  FY  1997. 

d.  Documentation  (1.2.3) 

This  category  covers  the  cost  of  producing  and  distributing  user’s  manuals  and 
training  materials  for  all  equipment  and  for  all  bases.  We  estimated  this  as  10  percent  of  the 
cost  of  application  software  investment.  This  does  not  include  the  cost  of  other  software 
documentation  such  as  requirements  specifications  and  design  documents;  those  are  already 
included  in  the  previous  estimates.  However,  it  does  include  a  one-time  charge  to  bring  the 
system  documentation  up  to  cunent  DoD  standards  (DOD-STD-7935).  This  latter  effort 
would  include  generating  various  deliverables  as  well  as  modifying  the  format  of  existing 
documentation.  We  are  estimating  a  total  of  10  staff-years  during  FY  1995  at  $132,480  per 
staff  year  for  a  total  of  $1.32  million. 
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Table  Vll-S.  Data 

Base 

Initialization  Costs 

Weapon  System 

Delta 

Initialization  Costs 
(Thousands  of  Etollars) 

F-I6C/D 

175 

Block  40 

0.1 

18 

Block  SO 

O.l 

18 

F-15 

145 

F-15A/B 

O.l 

15 

F-15C/D 

O.l 

15 

F-15E 

0.1 

15 

F-117 

80 

F-4 

275 

A-10 

275 

F-lll 

275 

EF-1 1 1 

0.5 

138 

B-1 

500 

B-2 

500 

B-52G 

275 

B-52H 

0.2 

55 

KC-10 

120 

KC-135E 

120 

KC-135R 

0.25 

30 

EC-135 

0.25 

30 

RC-135 

0.25 

30 

VC- 137 

0.25 

30 

T-IA 

100 

T-37 

100 

T-38 

100 

T-39 

100 

T-41 

100 

T-43 

100 

C-9 

160 

C-12 

160 

C-20 

160 

C-21 

160 

C-23 

160 

C-26 

160 

C-130E 

225 

C-130B 

0.1 

23 

C-130H 

0.1 

23 

HC-130N/P 

0.1 

23 

WC-130E/H 

O.l 

23 

MC-l30Em 

1 

22S 

AC-130A;H 

1 

225 

C-14I 

160 

E-3 

200 

E-4 

225 

E-8 

150 

H-1 

200 

H-3 

200 

H-53 

200 

MH-53 

1 

200 

H-60 

200 

MH-60 

1 

200 

UH-60 

0.1 

20 

Trainen/  Simulaton 

742 

Conun.-Electronics 

1.700 

ToUl 

9,858 
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e.  Communications  (1.3) 

(1)  Hardware  (1.3.1).  Communications  hardware  must  be  increased  to  support 
the  increased  number  of  daily  transactions.  This  increased  transaction  rate  may  be  handled 
by  adding  three  Memotec  High-Speed  Packet  Switch  central  processing  units  and  a 
memory  upgrade  to  each  of  two  DCP-30  processors.  The  cost  of  additional 
communications  hardware  is  $50,000. 

There  is  no  additional  nonrecurring  cost  for  communication  lines  because  Sprint 
lines  are  used  and  Sprint  does  not  charge  for  installation  on  new  lines. 

f.  Training/Unit  Activations  (1.4) 

Site  activation  is  carried  out  for  each  unit  or  squadron.  The  unit  activation  costs  are 
a  potentially  important  cost  driver  because  they  are  multiplied  254  times.  For  purposes  of 
the  analysis,  145  of  these  units  are  classified  as  high  complexity,  80  as  medium,  and  29  as 
low.  Activation  costs  were  calculated  by  identifying  the  steps  involved,  estimating  the 
number  of  staff-weeks  for  each  step  at  each  complexity  level,  and  multiplying  by  the 
number  of  units.  Our  estimates  of  the  resources  required  for  each  of  the  steps  were  based 
on  the  experience  at  Seymour  Johnson.  The  analysis  assumes  that  site  activation  will  be 
carried  out  over  a  19-month  period,  beginning  1  March  1996  and  ending  30  September 
1997.  For  each  of  the  following  steps,  we  assumed  that  39  percent  of  the  dollars  would  be 
spent  in  FY  1996  and  61  percent  in  FY  1997. 

(1)  Site  Surveys  (1.4.1).  Several  major  activities  are  involved  in  activating  a 
given  unit.  The  first  activity  is  to  conduct  a  site  visit  in  order  to  meet  with  unit  personnel,  to 
identify  all  terminals  and  printers  by  work  location,  and  to  assess  the  adequacy  of  the 
communications  facilities.  For  sites  that  do  not  have  access  to  external  communications,  a 
line  will  be  installed.  Sprint,  the  current  carrier  for  TICARRS,  does  not  charge  for 
installation.  We  assumed  that  each  unit,  regardless  of  complexity,  would  require  an  average 
of  40  hours  to  conduct  the  site  survey  at  the  $69  per  hour  rate.  This  implies  a  cost  of 
$2,760  per  unit.  The  total  across  all  254  units  is  $701,040. 

(2)  Load  CAMS  Data  (1.4.2).  A  second  major  activity  involves  loading  the 
CAMS  data  for  a  given  unit  into  TICARRS.  Some  CAMS  data  at  Seymour  Johnson  were 
found  to  be  in  error,  especially  in  terms  of  configuration.  TICARRS  has  configuration 
edits,  requiring  that  these  data  be  cleaned  up  before  they  can  be  entered  into  the  system.  In 
some  cases,  this  can  be  done  via  software;  in  other  cases,  it  has  to  be  done  manually,  lliese 
kinds  of  problems  are  likely  to  persist  for  each  unit  so  we  do  not  expect  them  to  decrease 
significantly  across  units.  The  cost  estimate  assumes  that  two  staff-months  (160  hours  per 
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month  or  320  hours  total)  per  unit  will  be  required  to  load  the  data  from  CAMS  for  high- 
complexity  units,  1.5  staff-months  for  medium-complexity  units,  and  1  staff-month  for 
low-complexity  units. 

The  cost  calculations  are  shown  below: 

145  units  X  320  hours  per  unit  =  46,400  hours  at  $69  =  $3,201,600, 

80  units  X  240  hours  per  unit  =  19,200  hours  at  $69  =  $1,324,800,  and 
29  units  X  160  hours  per  unit  =  4,640  hours  at  $69  =  $320,160 

The  total  is  $4,846,560, 

(3)  User  Training  (1.4.3).  Our  estimates  for  user  training  were  two  people  for 
three  weeks  per  unit  (240  hours),  regardless  of  complexity.  At  $40,25  per  hour,  this  comes 
to  $9,600  for  one  unit  and  $2,453,640  across  all  254  units. 

(4)  Resolve  DIREPS  (1.4.4).  There  were  approximately  150  Difficulty 
Reports  (DIREPs)  resulting  from  the  Assessment  at  Seymour  Johnson  or  an  average  of  50 
per  unit.  Approximately  135  of  these  resulted  in  changes  to  the  application  software  or  the 
data  base  (e.g.,  adding  a  specific  part  number/serial  number  combination),  or  in  resetting  of 
system  parameters  (e.g.,  to  allow  printing  to  a  desktop  printer).  These  DIREPs  averaged 
20  hours  each  to  research,  resolve,  and  re-test  the  affected  part  of  the  system  and  make  any 
necessary  changes  to  the  user  documentation.  Many  of  these  DIREPs  represent  “bugs”  in 
the  application  software  or  the  data  base.  We  expected  that  the  DIREPS  would  decline 
exponentially  for  any  given  weapon  system  over  units  (i.e.,  they  will  start  out  high  and 
then  drop  quickly).  This  steep  decline  is  typical  of  many  fielded  applications.  For  high- 
complexity  units,  we  assumed  an  average  of  15  DIREPS  per  unit.  We  assumed  an  average 
of  11  DIREPS  for  medium-complexity  units  and  an  average  of  8  for  low-complexity  units. 
The  breakout  is  as  follows: 

145  units  X  15  DIREPS  per  unit  x  20  hours  at  $69/hour  s  $3,001,500, 

80  units  X  1 1  DIREPS  per  unit  x  20  hours  at  $69/hour  =  $1,214,400,  and 
29  units  X  8  DIREPS  per  unit  x  20  hours  at  $69/hour  =  $320,160. 

The  total  across  all  254  units  is  $4,536,060. 

(5)  Short-Term  Support  (1.4.5).  Extra  support  from  DRC  site 
representatives  must  be  available  during  the  first  few  weeks.  We  are  assuming  one  extra 
site  representative  (at  $40.25  an  hour)  per  unit  for  three  weeks.  This  yields  a  total  of 
$1,229,868. 
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Travel  and  Living  (Site  Visit).  Travel  and  living  expenses  for  the  site  visits 
were  calculated  using  $100  day  for  5  days  per  unit.  This  comes  to  a  total  of  $127,000. 

(6)  Travel  and  Living  (Training)  (1.4.6).  Travel  and  living  expenses  for 
the  initial  training  were  calculated  using  $100  day  for  three  weeks  for  two  people.  This 
comes  to  $4,200  per  unit  or  $1,066,800  total. 

(8)  Travel  and  Living  (Short-Term)  (1.4.7).  Travel  and  living  expenses 
for  the  short-term  support  are  estimated  for  each  unit  at  $100  a  day  for  three  weeks.  The 
total  cost  for  254  units  is  $533,400. 

g.  Support  OT&E  (1.5) 

This  category  covers  support  for  an  operational  test  and  evaluation  in  preparation 
ior  the  MAISRC  Milestone  HI.  We  estimated  a  total  of  $2.5  million  spread  out  equally  over 
FY  1995  and  FY  1996. 

2.  Recurring  Costs  (2.0) 

a.  Program  Management  (2.1) 

(1)  Government  (2.1.1).  Costs  for  the  government  program  office  were 
estimated  to  be  10  percent  of  total  contractor  labor  (including  contractor  program 
management). 

(2)  Contractor  (2.1.2).  Costs  for  contractor  program  management  were 
estimated  to  be  5  percent  of  total  contractor  labor. 

b.  Computer  Operations  (2.2.) 

(1)  Central  Computer  (2.2.1).  The  cost  estimates  associated  with  the  central 
computer  are  broken  down  into  computer  usage,  operational  staff,  and  hardware,  software, 
and  utilities. 

Computer  Usage.  The  existing  computer  installed  at  DRC  will  be  used  to 
continue  the  current  support  provided  for  the  F- 16  and  other  aircraft.  This  cost  will  be 
terminated  with  the  purchase  of  computers  for  TICARRS  in  1996. 

Operational  Staff.  The  central  computer  operational  staff  cost  includes  all  the 
personnel  required  to  operate  the  central  computer  systems,  network  management,  technical 
support,  and  clerical  help  for  a  three-shift-per-day  operation.  This  is  estimated  to  be  33 
people  at  $77,280  per  year  for  an  annual  cost  of  $2,550,240  once  TICARRS  is  deployed. 
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Hardware,  Software,  and  Utilities.  The  cost  for  hardware  maintenance  and 
software  licensing  is  estimated  to  be  25  percent  of  the  cost  of  the  system  hardware. 

(2)  Base  Communications  Operations  (2.2.2,  2.2.3,  and  2.2.4).  This 
is  the  allocated  cost  for  maintenance  and  support  of  the  base-level  terminals  and 
communication  equipment.  The  annual  cost  is  the  same  as  that  allocated  to  CAMS,  once 
TICARRS  is  fully  deployed.  The  cost  to  TICARRS  at  the  end  of  the  measurement  period  is 
less  than  CAMS  because  TICARRS  does  not  start  to  provide  service  until  1996. 

The  base  communications  operations  costs  are  identified  by  CONUS,  USAFE,  and 
PACAF  organizations.  These  are  the  same  costs  as  were  applied  to  CAMS.  (See  cost 
description  for  CAMS  for  more  information.) 

Civilian  and  Military  Personnel.  This  is  the  cost  of  personnel  to  operate  and 
maintain  the  Data  Communication  Processor  (DCP),  communication  equipment,  and 
terminals  at  each  of  the  bases  allocated  to  TICARRS. 

Hardware,  Software,  Utilities.  This  is  the  estimated  cost  of  maintenance  of 
the  base  level  communication  and  terminal  equipment  allocated  to  TICARRS. 

c.  User  Support  (2.3) 

User  support  for  TICARRS  takes  the  form  of  site  representatives  residing  at  the 
bases.  We  estimated  one  site  representative  for  each  of  89  bases  by  the  end  of  FY  1997. 
The  cost  estimate  begins  with  34  at  the  beginning  of  FY  1996  (the  current  number)  and 
increases  to  89  through  FY  97.  We  also  included  two  site  representatives  for  each  of  the 
five  ALCs  (for  a  total  of  ten)  beginning  in  FY  1996.  Assuming  a  fully-burdened  hourly  rate 
of  $40.25  and  1,920  hours  per  year,  the  annual  cost  for  the  site  representatives  was 
estimated  as  follows: 

•  FY  1994  (34  site  representatives)  =  $2,627,520, 

•  FY  1995  (34  site  representatives)  =  $2,627,520, 

•  FY  1996  (average  of  59  site  representatives  +  10  ALCs)  =  $5,332,320, 

•  FY  1997  (average  of  84  site  representatives  10  ALCs)  =  $7,264,320,  and 

•  FY  1998  (straight-lined  at  89  site  representatives  +  10  ALCs)  =  $7,650,720. 

We  have  also  included  three  full-time  personnel  to  staff  a  help  desk  at  Andover  beginning 
in  FY  1996  at  $69  per  hour  for  a  total  of  $397,440. 
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d.  Communications  (2.4) 

The  estimated  yearly  communication  line  charges  are  $1.67  million  for  lines  linking 
the  Air  Force  bases  with  the  TICARRS  system  at  Andover,  Massachusetts. 

Our  estimate  is  based  on  a  total  of  89  bases  (66  CONUS  and  23  OCONUS),  57  of 
which  are  in  communication  now  (45  CONUS  and  12  OCONUS).  The  CONUS  bases  will 
need  to  be  upgraded  from  a  9.6-kilobytes  per  second  (Kbps)  to  a  19.2-Kbps  line.  The 
remaining  32  bases  (21  CONUS  and  1 1  OCONUS)  have  no  communication  links  and  will 
need  19.2-Kbps  lines.  In  addition,  Andover  (DRC)  will  require  two  additional  56-Kbps 
lines  to  handle  additional  traffic.  (Fifteen  F-16  and  F-I5  AFR  and  ANG  bases  now  share 
active  Air  Force  facilities.) 

The  costs  to  upgrade  from  9.6  Kbps  to  19.2-Kbps  is  $430  per  month.  Sprint  rates 
are  as  follows: 

•  a  19.2-Kbps  line  is  $1,145  per  month  and 

•  a  56-Kbps  line  is  $3,500  per  month. 

The  calculation  of  the  CONUS  plus  OCONUS  communications  costs  involves; 

•  45  CONUS  and  12  (X^ONUS  sites  at  $470  per  month:  $32 1 ,480  per  year, 

•  21  CONUS  and  1 1  (XONUS  sites  at  $1,145  per  month  $439,680  per  year, 

•  two  lines  at  $3,500  per  month:  $84,000  per  year, 

•  current  annual  cost  of  57  sites:  $562,992  for  1994  and  1995  only,  and 

•  total  projected  annual  communication  costs;  $  1 ,408, 1 52  per  year. 

This  cost  is  fully  realized  by  1998  when  TICARRS  is  fully  deployed. 

e.  Software  Maintenance  (2.5) 

(1)  Applications  Software  (2.5.1).  The  dollar  values  shown  for  FY  1996 
and  FY  1997  do  not  include  the  cost  of  resolving  DIREPS  that  are  reported  during  site 
activation;  those  were  included  above.  The  maintenance  assumed  in  this  cost  element 
involves  the  steady-state  tasks  of  responding  to  other  DIREPS,  development  of  minor 
enhancements,  and  so  on. 

The  estimate  assumes  that  25,000  lines  of  code  can  be  maintained  per  staff  year. 
TICARRS  is  currently  417,000  lines  of  unique  COBOL  code  (including  declarations  and 
executable  lines,  excluding  comments).  We  estimated  17  staff  years  for  FY  1994  through 
FY  1996. 
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Once  the  functional  enhancements  are  in  place,  those  lines  of  code  must  be 
maintained  as  well.  We  assumed  two  additional  maintainers  for  the  additional  39,000  lines 
of  programmer-generated  code  for  a  total  of  19.  At  $132,480  per  burdened  staff-year,  we 
estimated  $2,517,120.  We  then  added  10  percent  for  maintenance  of  the  Middleware  tables 
for  a  total  of  $2,768,832  per  year. 

(2)  Data  Base  Management  (2.5,2).  This  task  involves  the  changing  of  the 
data  base  structure  for  performance  purposes,  balancing  the  distribution  of  data  across  the 
data  storage  devices,  and  repairing/correcting  the  data  base  in  the  case  of  damage  by  a 
software  defect.  This  also  includes  maintaining  current  configuration  data  for  each  weapon 
system.  The  study  team  estimated  this  effort  would  require  8  experienced  data  base 
managers  in  FY  1994,  growing  to  14  in  FY  1996  and  to  20  in  FY  1997  for  a  total  of 
$2,649,600  per  year. 

(3)  Documentation  (2.5.3).  The  recurring  documentation  costs  represent 
changes  that  must  be  made  to  documentation  whenever  modifications  are  made  to 
TICARRS  that  impact  the  users.  These  could  be  modifications  as  a  result  of  resolving  a 
DIREP  or  modifications  that  represent  functional  enhancements.  They  have  been  estimated 
at  5  percent  of  the  cost  of  application  software  maintenance. 

f.  Training  and  Travel  (2.6) 

This  category  covers  the  cost  of  periodic  training  of  the  site  representatives  (who,  in 
turn,  conduct  the  user  training)  and  miscellaneous  travel  to  and  from  Andover  to  the  user 
sites.  These  costs  were  estimated  at  10  percent  of  the  cost  of  application  software 
maintenance. 

g.  Continuity  of  Operations  (2.7) 

This  cost  item  provides  TICARRS  with  the  capability  to  re-establish  operation  at 
another  site  in  the  event  of  a  catastrophic  failure  at  the  TICARRS  computer  center.  For  a 
monthly  fee,  Dataguard  Recovery  Services,  in  Louisville,  Kentucky,  would  guarantee 
TICARRS  with  access  to  an  off-site  compatible  computer  system  and  communication 
facilities  in  the  event  of  a  disastrous  outage  at  the  TICARRS  central  site.  The  monthly  fee 
provides  for  72  hours  testing  of  the  continuity  procedures  per  year.  Actual  operational 
usage  is  not  included  in  the  fee  except  as  part  of  the  continuity  test.  The  monthly  fee  quoted 
for  the  TICARRS  Bull  equipment  configurations  is  $14,600.  (See  Chapter  V  for  more 
information  about  continuity  of  operations). 
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F.  COST  DRIVERS 


The  cost  differences  that  result  from  comparing  CAMS/REMIS  to  TICARRS  can  be 
understood  by  identifying  and  evaluating  the  cost  drivers  for  the  information  systems.  The 
following  factors  greatly  affect  costs  in  maintenance  information  systems: 

•  volume  of  application  software  (developed,  maintained)  and  size  and 
complexity  of  the  data  base. 

•  computer  operations, 

•  user  support  (both  on  and  off  the  site),  and 

•  communications. 

The  cost  driver  that  affects  volume  of  software  is  measured  by  lines  of  codes  that 
must  be  generated  and  maintained  by  the  programmers.  The  other  three  factors  are  heavily 
influenced  by  the  basic  architecture  of  the  systems  (e.g.,  central  data  base  versus 
distributed  system  versus  a  base-level  replicated  system). 

Table  Vn-9  shows  a  representative  annual  recurring  costs  for  CAMS,  REMIS,  and 
TICARRS  that  can  be  attributed  to  the  four  major  cost  factors  cited. 


Table  VII-9.  Comparison  of  Costs  Across  Systems 


Maior  Factors 

Cost  in  Millions  of  Dollars 

CAMS 

REMIS 

CAMS/REMIS 

TICARRS 

Volume  of  Software 

$3.5 

$5.6 

$9.1 

$5.7 

Ccxnputer  Operations 

$10.1 

$2.4 

$12.5 

$4.0 

User  Support 

$18.4 

$1.1 

$19.5 

$8.1 

Communications 

$0.2 

$13.7 

$13.8 

The  table  clearly  shows  that  the  costs  for  CAMS/REMIS  in  three  of  the  categories 
exceed  those  for  TICARRS.  For  volume  of  software,  this  result  is  partly  attributable  to  the 
use  of  Middleware  in  TICARRS,  which  has  the  significant  effect  of  lowering  the 
programmer-generated  and  programmer-maintained  lines  of  code  relative  to 
CAMS/REMIS.  Moreover,  CAMS  and  REMIS  are  really  two  systems,  requiring  a 
significant  amount  of  computer  code  that  fosters  interfaces  between  them.  That  would  be 
unnecessary  in  one  system  containing  the  same  total  functionality. 

For  computer  operations  and  user  support,  the  cost  differences  result  from  the  level 
of  effort  required  to  develop,  support,  and  maintain  a  central  data  base  architecture  system 
such  as  TICARRS,  relative  to  the  base-level  replicated  system  of  CAMS  combined  with 
cost  of  supporting  a  separate  REMIS  central  data  base.  Much  of  what  is  done  at  each  base 
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for  CAMS  can  be  centrally  controlled  and  efficiently  undertaken  at  the  TICARRS  central 
site.  For  example,  whereas  both  CAMS  and  TICARRS  require  one  site  representative  for 
typical  user  support,  CAMS  also  requires  two  to  three  data  base  managers  per  base  to 
undertake  activities  at  the  base  that  are  performed  only  once  fleet-wide  at  the  TICARRS 
central  site. 

The  key  cost  driver  is  the  system  architecture — central  data  base  versus  one  that  is 
replicated  for  each  base.  For  computer  operations  and  user  support,  this  cost  driver  results 
in  a  cost  difference  of  almost  $20  million  annually.  In  addition,  because  Middleware 
reduces  the  value  of  programmer-operated  and  -maintained  code,  associated  costs  are 
reduced. 

G .  COST  COMPARISON  OF  ALTERNATIVES 

1 .  Alternative  1:  Enhanced  Version  of  CAMS/REMIS 

In  this  alternative,  CAMS/REMIS  would  continue  with  the  enhancements  discussed 
previously.  TICARRS  would  be  phased  out  beginning  at  the  time  of  FOC  of  REMIS 
(projected  for  January  1995).  Since  the  weapon  systems  supported  by  TICARRS  are  also 
supported  by  REMIS,  we  assumed  that  the  phasing  out  could  be  done  quickly.  For  our 
analysis,  the  phase-out  would  be  done  over  a  three-month  period  (January  through  March 
1995).  Thus,  TICARRS  would  operate  throughout  FY  1994  and  through  the  first  six 
months  of  FY  1995. 

In  this  alternative,  TICARRS  would  not  be  expanded  but  would  continue  with  its 
current  scope  and  functionality.  To  estimate  the  costs  of  operating  TICARRS  throughout 
FY  !994  and  the  first  six  months  of  FY  1995,  we  used  the  recurring  cost  estimates  from 
Table  VII-5,  The  total  for  TICARRS  for  FY  1994  is  $1 1,270,000.  We  assumed  50  percent 
of  this  amount  would  cover  the  first  six  months  of  FY  1995. 

The  costs  of  CAMS  and  REMIS  in  this  alternative  arc  equivalent  to  the  costs 
estimated  for  them  earlier  in  this  chapter  and  displayed  in  Tables  VII- 1  and  Vn-4.  Table 
VII-10  shows  the  costs  of  Alternative  1. 

2 .  Alternative  2:  Enhanced  Version  of  TICARRS 

In  the  second  alternative,  TICARRS  would  be  expanded  in  terms  of  functionality 
and  scope  as  already  described.  We  estimated  that  TICARRS  would  not  be  completely 
activated  until  tte  end  of  FY  1997.  REMIS  would  continue  operating  until  then.  Air  R)rce 


Vn-37 


units  would  be  transferred  from  CAMS  to  TICARRS  during  FY  1996  and  FY  1997.  Table 
Vn-1 1  presents  the  cost  estimates  for  Alternative  2. 

The  following  subsection  discusses  the  changes  to  CAMS  expenditures  under  this 
alternative. 

a.  Cost  Estimate  for  CAMS  Under  Alternative  2 

With  this  alternative,  the  CAMS  cost  estimates  that  were  shown  in  Table  VII- 1  will 
be  changed  substantially.  Table  VII- 12  shows  the  revised  cost  estimate  for  CAMS. 

Unless  otherwise  noted,  recurring  CAMS  costs  for  FY  1996  and  FY  1997  are 
proportional  to  the  phasing  in  of  TICARRS  during  FY  1996  and  FY  1997.  During  FY 
1996,  39  percent  of  the  sites  are  activated  and  61  percent  in  FY  1997.  Remaining  CAMS 
costs  are  80  and  30  percent,  respectively,  of  FY  1994  costs. 

(1)  Nonrecurring  Costs 

CAMS  Share  of  RPC  Hardware.  We  assumed  that  if  Alternative  2  is  chosen, 
purchases  of  hardware  for  the  RPCs  would  continue  uninterrupted.  The  DoD  is 
establishing  Megacenters  to  centralize  and  standardize  selected  computer  operations  across 
DoD.  Each  RPC  is  slated  to  change  over  to  a  Megacenter.  That  portion  of  RPC  computing 
capability  dedicated  to  CAMS  would  become  available  to  support  Megacenter  operating 
requirements.  In  FY  1998,  when  TICARRS  is  fully  operational,  there  will  be  savings 
equivalent  to  the  CAMS  portion  of  RPC  computer  hardware.  The  savings  were  calculated 
as  the  replacement  value  of  the  equipment  in  the  year  that  the  change  to  TICARRS  is 
completed. 

Software.  CSRDs  would  be  retained  for  completion  under  Alternative  2.  We 
assumed  that  further  investments  for  enhanced  editing  to  improve  CAMS  would  be  halted. 
Expenditures  for  system  integration  and  test  to  support  the  continued  CSRD  initiatives 
would  remain. 

(2)  Recurring  Costs.  Under  Alternative  2,  there  would  be  a  number  of  impacts 
on  recurring  costs  as  well.  All  cost  elements  would  disappear  after  FY  1997.  For  most  of 
the  cost  elements,  there  would  be  changes  in  the  earlier  years  as  well.  These  are  discussed 
below. 

Program  Management.  We  assumed  that  program  management  costs  would 
remain  unchanged  proportionally  under  Alternative  2  until  FY  1996  and  then  decline 
through  FY  1997. 
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Tabl«  Vil'10.  Cost  Estimate  for  Alternative  1:  Enhanced  Version  of  CAMS/REMIS 
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Operations.  For  both  RPC  operations  and  base  communications  operations,  as 
CAMS  is  phased  out  over  the  FY  1995-1997  period,  the  cost  of  operating  the  mainframes 
would  decrease  as  well.  We  assumed  that  the  funding  allocated  to  CAMS  during  FY  1994 
and  FY  1995  would  remain  at  that  level  since  no  units  would  be  moved  from  CAMS  to 
TICARRS  during  that  period.  We  assumed  a  decrease  consistent  with  TICARRS 
activations  throughout  1996  and  FY  1997  at  the  RPCs  in  CONUS  and  overseas.  These 
expenditures  are  identified  in  cost  elements  2.2,1  through  2.2.6  in  Table  vn-1. 

User  Support.  User  support  would  continue  until  FY  1996  as  planned  and  then 
declines  as  units/bases  are  converted  to  TICARRS. 

Communications.  We  assumed  that  the  high-speed  (52  Kbps)  lines  would 
remain  in  place  for  the  FY  1994-1995  period  and  then  decline  through  FY  1996  and  FY 
1997  as  the  transition  to  TICARRS  occurs. 

High-Speed  REMIS  Link.  This  link  is  not  required  under  Alternative  2. 

Software  Maintenance.  The  number  of  people  engaged  in  software 
maintenance  (including  applications  software,  system  integration  and  test,  data  base 
management,  and  documentation)  at  Gunter  SSC,  would  be  reduced  in  FY  1996  and 
FY  1997. 

Training  and  Travel.  Costs  for  FY  1996  and  FY  1997  would  decline  as  the 
transition  to  TICARRS  occurs. 

b.  Cost  Estimate  for  REMIS  Under  Alternative  2 

The  REMIS  costs  are  adjusted  to  reflect  the  termination  of  the  program  in  FY  1998 
and  reduced  scope  of  work  in  FY  1996  and  FY  1997  as  TICARRS  is  activated.  REMIS 
costs  are  shown  in  Table  VII- 13.  Nonrecurring  costs  are  included  to  complete  OT&E, 
improve  performance  and  REMISTALK  at  a  reduced  level  of  effort,  update  documentation, 
and  provide  training  on  GCSAS.  This  will  allow  the  Air  Force  to  complete  the  process  of 
having  REMIS  replace  several  information  systems  beginning  in  FY  1995.  The  recurring 
costs  reflect  full  operations  of  REMIS  in  FY  1994  and  FY  1995  with  support  being 
continued  at  a  substantial  level  in  IT  1996  and  FY  1997. 
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_ _ Cost  in  Millions  of  1994  Dollars _ 

Category  of  Cost _ 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  Total 

pecurring  Costs 
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3.  Comparison  of  Alternatives 

Table  VII-14  shows  that  the  TICARRS-based  alternative.  Alternative  2  was 
estimated  to  have  lower  ten-year  costs  than  was  the  alternative  based  on  CAMS/REMIS.  In 
present  value  (discounted)  terms.  Alternative  2  is  $100  million  cheaper,  a  savings  of 
18  percent  relative  to  Alternative  1. 

4 .  Sensitivity  Analysis 

In  Alternative  2,  a  four- year  transition  period  was  assumed  for  TICARRS. 
However,  there  is  potential  for  delays  in  both  the  acquisition  process  and  in  the 
development  program.  The  effect  on  the  cost  of  the  TICARRS  alternative  if  the  program  is 
delayed  one  year  is  reviewed  in  the  following  discussion. 

To  assess  the  costs  to  the  TICARRS  program  schedule  for  a  one-year  delay,  we 
have  assumed  that  the  contract  award  would  be  delayed  for  six  months  and  the 
development  program  would  take  an  additional  six  months.  This  has  the  effect  of  delaying 
the  full  activation  of  TICARRS  until  FY  1999.  Table  VIII-15  shows  the  impact  of  the  one- 
year  delay  on  TICARRS  costs. 

Using  a  similar  methodology  to  that  presented  in  Alternative  2,  we  estimated  the 
costs  of  extending  the  operations  of  CAMS  and  REMIS  by  one  year.  Our  revised  estimates 
of  CAMS  and  REMIS  costs  are  shown  in  Tables  VII-16  and  VII-17,  respectively.  Table 
VII- 18  shows  our  revised  cost  estimates  for  Alternative  2  with  the  one-year  delay  in 
fielding  TICARRS. 

Over  the  ten-year  period,  the  present  value  of  total  costs  to  implement  TICARRS  in 
five  years  is  $485.5  million,  compared  with  $562.3  million  for  the  CAMS/REMIS 
alternative  (Alternative  1).  This  represents  a  savings  of  $76.8  million  or  14  percent 

Table  VII- 19  presents  a  comparison  of  the  costs  for  Alternative  1  and  the  revised 
Alternative  2. 
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Table  VII-14.  Comparison  of  Alternatives 

Millions  of  1994  Dollars _ 

of  Cost  1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  Total 
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Tabto  Vli-15.  Revisad  Cost  Estimate  for  TICARRS  Under  Alternative  2  With  One-Year  Delay  in  TICARRS 

_ _ _ Cost  in  Millions  _ 

_ Category  of  Cost _ 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  Total 

1.0  Nonrecuirtng  Costs 
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TabI*  VII-15.  Ravised  Cost  Estimate  for  TICARRS  Under  Alternative  2  With  One-Year  Delay  in  TICARRS 

(Continued) 

_ Cost  in  Millions _ 
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Haidwaie,  Software,  and  Utilities  2.88  2.88  2.45  1.77  0.69  10  67 

2.2.2  RPC  PACAF  Operations  0.00 

Direct  Pfcisonnel  0.’2  0.12  0.12  0.12  0.12  0.59 

Shared  Personnel  0.65  0.65  0.65  0.65  0.65  3.27 

Hardware.  Software,  and  Utilities  0.51  0.51  0.43  0.39  0.41  2.25 
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VIII.  CONCLUSIONS 


This  paper  describes  IDA’s  comparison  of  the  effectiveness  and  costs  of  two 
alternative  Air  Force  maintenance  information  systems,  CAMS/REMIS  and  TICARRS. 
Two  formal  alternatives  were  defined  and  reviewed  in  detail.  We  evaluated  CAMS/REMIS 
and  TICARRS  as  they  exist  today  (considering  current  plans  and  available  budgets). 
CAMS/REMIS  is  the  standard  Air  Force  maintenance  information  system.  TICARRS  is 
used  to  provide  broad  direct  support  to  the  F-16  and  F-1 17  communities  and  serves  many 
weapon  system  management  functions  for  the  F-1 5  and  F-16. 

We  compared  the  systems  along  six  dimensions  of  effectiveness;  system 
functionality  (what  they  do),  scope  (for  what  systems  and  equipment),  operating 
characteristics  (their  availability,  responsiveness,  and  ease  of  use),  data  accuracy  and 
completeness,  adaptability  (their  ability  to  respond  quickly  to  changes  in  user  needs  and  to 
ease  the  transition  to  the  next  generation  maintenance  information  system),  and  logistics 
and  operational  effectiveness  (measured  by  mean  time  between  failures,  maintenance  man¬ 
hours  per  flying  hour  and  mission-capable  rates).  Limitations  of  the  systems  were 
identified  through  a  review  of  available  documentation  and  extensive  discussions  with 
u.sers  and  system  developiers.  In  addition,  IDA  was  able  to  use  the  Operational  Assessment 
of  TICARRS  that  took  place  at  Seymour  Johnson  AFB  during  the  spring  of  1993  to  better 
understand  some  of  its  particular  strengths  and  shortcomings.  Because  there  were  relatively 
few  REMIS  users  and  REMIS  has  not  yet  achieved  its  full  operating  capability,  we 
conducted  a  special  test  of  its  functionality  and  selected  operating  characteristics.  To  the 
extent  possible,  steps  required  to  overcome  the  limitations  of  each  system  were  specified 
and  cost  estimates  developed. 

The  ten-year  costs  of  the  systems,  modified  as  needed  to  improve  their 
effectiveness,  were  then  compared.  A  ten-year  time  period  was  used  because  it  seems  likely 
that  newer  information  system  technology  should  displace  whatever  system  is  chosen  in 
about  ten  years. 

The  remainder  of  this  chapter  summarizes  the  major  findings  of  our  work. 
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A.  EVALUATION  OF  EXISTING  SYSTEMS 


Table  VIII- 1  summarizes  IDA’s  findings  regarding  the  effectiveness  of 
CAMS/REMIS  and  TICARRS  as  they  exist  today. 


Table  Vill-1.  Comparison  of  the  Effectiveness  of  CAMS/REMIS  and  TICARRS 


Dimension  of  Effectiveness 

Conclusion 

Functionality 

Greater  for  CAMS/REMIS 

Scope 

Operating  characteristics 

Greater  for  CAMS/REMIS 

Availability 

Better  for  TICARRS 

Responsiveness 

Better  for  TICARRS 

Ease  of  use 

Equal  at  wing  level,  TICARRS  better  elsewhere 

Data  accuracy  and  completeness 

Better  for  TICARRS 

Adaptability 

TICARRS  better  in  short-term;  little  long-term 
difference 

Logistics  and  operational  effectiveness 

No  difference  found 

1 .  Functionality 

Today,  CAMS/REMIS  has  greater  functionality,  although  there  are  some  functions 
for  which  TICARRS  is  unique  (e.g.,  block  number  tracking).  Among  other  things,  CAMS 
provides  interfaces  with  the  Standard  Base  Supply  System  and  the  Comprehensive  Engine 
Management  System,  and  handles  personnel  and  training  management.  Gaps  in  the 
functionality  of  TICARRS  were  identified  during  the  Operational  Assessment  at  Seymour 
Johnson  AFB.  Relatively  Conclusionsg  the  Operational  Assessment.  Important  ones  that 
were  not  eliminated  included:  production  scheduling  and  management  (provision  of 
maintenance  snapshots),  appropriate  handling  of  PQDRs,  and  the  means  to  provide  a  14- 
day  records  check  in  an  Air  Force-approved  manner. 

2 .  Scope 

CAMS/REMIS  currently  supports  far  more  aircraft  and  other  products 
(communications-electronics  equipment,  AGE,  etc.)  than  does  TICARRS.  If  TICARRS 
were  to  replace  CAMS/REMIS,  it  would  have  to  be  modified  to  address  more  than  40 
additional  aircraft  types  and  1,700  types  of  communications  and  electronic  equipment 

3.  Operating  Characteristics 

Overall,  TICARRS  has  substantially  better  operating  characteristics  than  does 
CAMS/REMIS.  These  are  briefly  described  in  the  following  paragraphs. 


a.  Availability 

Regarding  system  performance  parameters.  TICARRS  and  REMIS  provide  very 
high  levels  of  availability  when  compared  to  CAMS.  TICARRS  and  REMIS  both  have  had 
less  than  one  percent  of  unscheduled  downtime.  CAMS  has  been  unable  to  approach  its 
target  of  95  percent  availability  at  the  vast  majority  of  bases. 

b.  Responsiveness 

TICARRS  has  better  response  time,  panicularly  compared  to  REMIS.  Some 
evidence  indicated  that  TICARRS  was  able  to  produce  standard  reports  in  less  than  30 
minutes,  while  REMIS  averaged  2.4  hours.  Ad  hoc  REMIS  reports  took  an  average  of  7.5 
hours  and  in  some  cases  up  to  36  hours. 

c.  Ease  of  Use 

Regarding  ease  of  use,  neither  system  is  convincingly  superior  at  the  wing  level. 
Discussions  with  a  wide  range  of  wing-level  users  uncovered  more  enthusiasm  for 
TICARRS  than  for  CAMS,  but  surveys  conducted  during  the  Operational  Assessment  at 
Seymour  Johnson  AFB  judged  the  two  systems  roughly  equal.*  Other  users  (depots, 
MAJCOM,  weapon  system  SPOs,  and  contractors)  emphasized  problems  with  REMIS — 
with  significant  exceptions  at  the  depot  level — and  exhibited  a  preference  for  TICARRS 
when  they  were  familiar  with  it. 

4 .  Data  Accuracy  and  Completeness 

Where  comparisons  are  possible,  TICARRS  has  been  shown  to  have  more 
complete  and  accurate  data.  Over  a  quarter  of  avionics  parts  arriving  for  depot-level  repair 
have  not  had  a  maintenance  action  appropriately  recorded  in  CAMS.  An  examination  of 
serially-controlled  parts  on  F-16  aircraft  undergoing  phase  inspections  found  that  between 
20  and  40  percent  of  the  changes  made  between  periodic  inspections  were  not  recorded  in 
CAMS.  This  compares  with  a  level  of  6  percent  in  TICARRS  when  it  was  supporting  F-16 
maintenance  at  the  wing  level. 


*  Mter  four  weeks,  users  exhibited  a  clear  preference  for  TICARRS.  At  six  weeks,  more  satisfaction  was 
expressed  for  CAMS  Throughout  the  assessment,  a  substantial  percentage  of  users  were  indifferent  to 
the  information  system.  The  fact  that  the  Operational  Assessment  was  limited  to  six  weeks  must  be 
considered  in  attempting  to  assess  the  ease  of  use  of  TICARRS.  A  longer  perfmmance  period  may 
result  in  very  different  qrinions. 
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REMIS  has  experienced  wide-spread,  persistent  failure  to  receive  the  data  it  is 
supposed  to  get  from  CAMS.  A  study  comparing  LRU  removals  in  REMIS  and  TICARRS 
found  no  data  from  the  REMIS  standard  query  and  only  minimal  data  in  REMISTALK. 
Only  the  TICARRS  data  seemed  realistic. 

REMIS  has  been  rejecting  between  8  and  11  percent  of  the  aircraft-related 
information  (which  represents  one  of  the  most  critical  types  of  equiiiment  tracked  in 
REMIS)  it  receives  from  CAMS  because  of  data  errors.  For  all  equipment  types,  the  total 
reject  rate  has  ranged  between  16  and  21  percent^ 

5 .  Adaptability 

TICARRS  is  more  adaptable  in  some  ways.  The  Operational  Assessment  at 
Seymour  Johnson  showed  that  minor  modifications  to  the  system  could  be  made  quickly. 
TICARRS  appears  to  be  closer  than  CAMS  to  being  able  to  deploy  the  kind  of  stand-alone 
system  needed  for  wartime. 

Neither  system  is  likely  to  have  a  distinct  advantage  in  aiding  the  transition  to  the 
next  generation  of  maintenance  information  systems. 

6.  Logistics  and  Operational  Effectiveness 

We  found  no  statistically  significant  relationships  between  the  choice  of 
maintenance  information  system  and  indicators  of  aircraft  readiness  and  supportability 
(mission-capable  rate,  mean  time  between  failures,  and  maintenance  man-hours  per  flying 
hour).  This  is  a  short-run  result,  and  does  not  address  the  issue  of  using  the  information 
from  these  systems  in  longer-term  efforts  to  identify  bad  actors,  justify  modifications,  and 
so  on. 

7 .  Summary 

CAMS/REMIS  now  handles  a  greater  number  of  weapon  systems  and  other  types 
of  equipment  than  does  TICARRS.  It  also  has  several  important  functions  that  are  missing 
from  TICARRS.  However,  TICARRS  operates  better,  and  has  the  inherent  capability  to 
provide  more  accurate  and  complete  data. 


^  This  may  reflect  problems  with  CAMS  data,  not  REMIS’s  ability  to  produce  valid  edit 
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B .  THE  ABILITY  OF  THE  SYSTEMS  TO  OVERCOME  THEIR 
SHORTCOMINGS 

In  deciding  which  maintenance  information  system  could  best  fill  the  Air  Force’s 
requirements  over  the  next  ten  years,  one  must  consider  the  ability  of  the  systems  to 
overcome  their  shortcomings. 

A  project  is  under  way  to  improve  some  of  CAMS’s  operating  characteristics.  The 
fielding  of  REMIS’s  GCSAS  subsystem  holds  the  promise  of  improving  the  accuracy  of 
CAMS  (and  thus  REMIS)  data.  REMIS  is  addressing  some  of  its  responsiveness 
problems.  Our  cost  estimates  include  some  additional  expenditures  to  further  improve  these 
characteristics. 

Given  the  amounts  of  money  dedicated  to  improving  the  performance  of  each 
system,  it  is  appropriate  to  assess  the  technical  risk  involved  in  successfully  completing  the 
required  system  improvements. 

In  our  judgment,  overall  risk  is  low  to  medium  for  the  CAMS/REMIS  alternative 
and  low  for  the  TICARRS  alternative. 

There  are  some  aspects  of  the  design  and  history  of  CAMS/REMIS  that  cast  doubt 
on  the  ability  of  that  system  to  significantly  improve  its  performance. 

The  move  to  Regional  Processing  Centers  will  not,  by  itself,  improve  the 
availability  of  CAMS.  Improvements  to  availability  are  dependent  upon:  (1)  improved 
software  quality  and  (2)  the  provision  of  the  appropriate  mix  of  skilled  personnel  (data  base 
managers)  at  the  bases.  Moreover,  CAMS  will  continue  to  be  operated  in  an  environment  in 
which  it  must  compete  with  other  base-level  applications  for  computer  system  resources. 
System  responsiveness  is  likely  to  continue  to  suffer  in  the  future. 

It  should  be  possible  to  adequately  improve  the  responsiveness  of  REMIS.  We 
believe  that  this  will  require  substantially  greater  resources  than  are  currently  being  applied. 

The  biggest  potential  problem  involves  the  completeness  and  accuracy  of  CAMS 
and  REMIS  data.  The  fielding  of  GCSAS  may  improve  the  situation,  but  the  complex 
nature  of  the  interface  between  the  two  systems,  and  the  data  rejects  that  may  still  result, 
could  result  in  data  that  are  not  accurate  enough  to  meet  some  of  the  Air  Force’s 
requirements.  It  is  difficult  enough  to  enforce  data  integrity  within  one  system. 
CAMS/REMIS  may  be  programmatically  one  system  but  they  are,  in  reality,  two  systems, 
with  different  record  formats  and  different  edits. 
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While  it  appears  that  TICARRS  is  capable  of  being  expanded  in  functionality  and 
scope  to  perform  the  tasks  that  CAMS/REMIS  currently  supports,  a  major  expansion  is  not 
without  risks.  In  our  analysis  of  alternative  systems,  we  included  the  cost  of  expanding 
TICARRS  and  changing  all  Air  Force  units  from  CAMS  to  TICARRS,  estimated  to  be  over 
$40  million.  Even  though  limitations  in  scope  and  functionality  are  weaknesses  of 
TICARRS  today,  the  flexibility  of  the  system’s  architecture  supports  optimism  about 
TICARRS’s  ability  to  be  expanded  as  needed. 

C.  COSTS 

1 .  Alternative  1:  Enhanced  Version  of  CAMS/REMIS 

We  estimated  that  the  alternative  based  on  CAMS/REMIS  would  cost  $663  million 
(in  FY  94  dollars)  over  the  next  ten  years  ($562  million  when  future  costs  are  discounted). 
The  major  cost  drivers  are  user  support  at  the  bases  and  computer  operations.^  Historically, 
CAMS  has  operated  with  an  average  of  three  to  four  data  base  managers  per  base,  and  we 
see  no  factors  that  would  significantly  change  that  level  in  the  future.  We  have  not 
attributed  any  costs  related  to  computer  equipment  that  has  already  been  purchased  for  the 
RPCs  to  CAMS. 

2 .  Alternative  2:  Enhanced  Version  of  TICARRS 

We  assumed  that  the  functionality  of  TICARRS  can  be  expanded  to  match  that  of 
CAMS/REMIS  by  the  end  of  FY  1995.  Adding  the  required  functionality  is  not,  in  our 
judgment,  a  major  cost  driver.  The  major  cost  drivers  associated  with  expanding  TICARRS 
are  in: 

•  initializing  the  data  base  for  each  new  weapon  system, 

•  acquiring  additional  hardware  to  handle  the  additional  work  load,  and 

•  changing  the  individual  Air  Force  units  from  CAMS  to  TICARRS. 

Moreover,  the  various  requirements  of  a  formal  MAISRC  acquisition  process  result  in 
TICARRS  being  fielded  in  place  of  CAMS/REMIS  no  earlier  than  FY  1997.  An  alternative 
based  on  TICARRS  was  estimated  to  have  a  ten-year  cost  of  $529  million  ($462  million 
when  discounted). 


^  The  CAMS  architecture  is  infarvently  expensive  because  it  is  replicated  at- all  bases,  even  with  the 
advent  of  the  RPCs.  We  considered  only  the  CAMS  portion  of  the  computer  costs  as  relevant  to  our 
analysis.  It  is  wtath  noting  that  CAMS’s  costs  wcHild  be  evmi  bigbo'  without  the  RPCs. 
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D.  OVERALL  CONCLUSIONS 


In  our  judgment,  an  alternative  based  on  implementation  of  TICARRS  provides  at 
least  as  great  effectiveness  with  less  risk  and  less  cost  than  CAMS/REMIS.  In  terms  of 
effectiveness,  CAMS/REMIS  now  suffers  from  problems  with  availability, 
responsiveness,  and  data  integrity.  TICARRS  is  currently  missing  some  important 
elements  of  functionality  and  cannot  support  as  many  systems  and  kinds  of  equipment  as 
can  CAMS/REMIS.  In  our  cost  analysis,  we  included  the  cost  of  additional  work  needed  to 
overcome  the  shortcomings  of  both  systems.  This  included  work  to  improve  CAMS  editing 
capability,  including  real-time  access  from  CAMS  to  REMIS  for  fleet-wide  information. 
We  remain  very  uncertain,  however,  that  the  data  integrity  problems  of  CAMS/REMIS  can 
be  fully  overcome.  Not  only  have  these  problems  been  persistent,  but  the  CAMS/REMIS 
architecture  works  against  their  simple  resolution. 

We  conclude  that  when  all  the  relevant  costs  are  considered,  a  system  based  on 
TICARRS  would  be  more  cost-effective  than  one  based  on  CAMS/REMIS. 

Assuming  a  four-year  phase-in  period,  we  estimated  that  the  Air  Force  could  save 
$100  million  in  present  value  terms  over  the  next  ten  years  by  choosing  a  TICARRS-based 
system.  This  is  18  percent  of  what  we  estimated  it  would  cost  to  operate  a  CAMS/REMIS- 
based  system  over  that  period.  If  the  policies  and  procedures  of  the  acquisition  process 
require  longer  to  change  over  to  a  TICARRS-based  system  (i.e.,  five  years),  we  estimated 
that  the  Air  Force  could  save  $77  million  in  present  value  terms  over  the  ten-year  period. 
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Data  accuracy  and  completeness  was  one  of  our  seven  dimensions  of  effectiveness. 
We  discovered  that  the  degree  of  accuracy  needed  varied  depending  on  the  application.  To 
illustrate  the  consequences  of  data  accuracy  in  one  application,  we  investigated  the 
relationship  between  the  detection  of  a  bad  actor  and  data  accuracy. 

The  detection  of  a  bad  actor  depends  upon  the  accuracy  and  completeness  of  the 
data  system  being  used  to  track  line  replaceable  units  (LRUs).  This  dependency  will  be 
illustrated  by  computing  the  expected  number  of  maintenance  cycles  that  an  LRU  is  in  the 
maintenance  system  before  it  is  detected  and  classified  as  a  bad  actor,  given  a  specific  level 
of  data  system  accuracy.  A  maintenance  cycle  is  defined  as  the  time  a  part  is  tested  and  fails 
to  the  time  it  is  tested  and  fails  again. 

A  study  on  the  relationship  between  data  accuracy  and  the  detection  of  bad  actors 
was  done  by  Dynamics  Research  Corporation.*  A  Markov  chain  approach  was  used.  We 
examined  that  analysis  and  made  some  modifications.  We  also  used  a  Markov  chain 
approach  to  determine  the  relationship  between  the  detection  of  a  bad  actor  and  data  system 
accuracy  and  completeness.  The  analysis  focused  on  a  particular  LRU’s  life  cycle, 
independent  of  its  delivery  to  the  Air  Force.  The  process  begins  with  a  set  of  operating 
aircraft  with  a  set  of  spares  (of  which  some  may  be  bad  actors).  We  made  the  following 
assumptions: 

•  A  bad  actor  is  defined  as  any  LRU  that  has  experienced  three  or  more  bench 
check  serviceable  (BCS)  actions  within  a  90-day  period.  To  declare  a  part  a 
bad  actor,  three  BCSs  need  to  have  been  recorded  in  the  data  system  within  a 
90-day  period. 

•  The  Air  Force  data  system  started  with  no  a  priori  knowledge  of  the  state  of  the 
part. 

•  A  maintenance  cycle  is  30  days  long. 


*  AUan  KJeinman,‘niie  Impact  of  Data  System  Accuracy  on  Detecting  Bad  Acum,”  Dynamics  Research 

Coqwration. 


•  Every  30  days,  a  BCS  takes  place,  along  with  an  opportunity  to  record  the 
observation  in  the  data  system.  The  observation  may  or  may  not  be  recorded, 
and  even  if  recorded,  the  possibility  of  error  exists; 

•  A  failure  occurred  at  time  t  =  0  and  the  opportunity  to  record  the  observation  is 
instantaneous. 

Thus,  it  is  possible  to  detect  a  bad  actor  (three  or  more  recorded  BCSs  within  a  90- 
day  period)  within  a  60-day  period.  It  is  also  possible  to  have  three  recorded  BCSs  with 
one  inaccurate  or  missed  record  between  them,  within  a  90-day  period.  Hence,  the  model 
needs  to  include  a  90-day  “window”  over  the  opportunities  to  record  the  BCS.  Note  that 
the  possibility  of  misidentifying  a  part  as  a  bad  actor  was  not  included  in  this  analysis. 
Further  investigation  would  have  needed  to  be  done. 

The  process  can  be  modeled  as  a  Markov  chain.  A  Markov  chain  is  a  stochastic 
process  that  takes  on  a  finite  or  countable  number  of  possible  values  or  states  and  exhibits 
the  Markovian  property  (i.e.,  the  probability  of  the  process  moving  to  a  future  state 
depends  only  on  the  present  state  and  not  on  past  states).  Also,  the  probability  of  moving 
from  state  to  state  is  not  dependent  on  time.  The  states  of  the  process  for  the  Hrst  case  we 
examined  are  shown  in  Figure  A-1. 


Figuro  A-1.  Markov  Chain  Stato  Diagram:  Caso  1 


The  states  of  the  Maikov  chain  are  labeled  with  letters  to  represent  the  results  of  the 
last  three  opportunities  to  record  bench  check  serviceables,  where  the  third  letter  repiesrats 
the  result  of  the  most  recent  observation  that  has  occurred,  the  second  letter  represents  the 
result  of  the  second  most  recent,  and  the  first  letter  represents  the  third  most  recent  The 
results  fall  into  two  classes:  (1)  the  data  was  accurately  recorded,  denoted  as  H  (“Hh”),  or 
(2)  any  other  possibility  (not  recorded,  recorded  inaccuratdiy,  etc.),  denoted  as  M  (“Miss”). 
For  example,  the  state  MHH  represents  two  observations  accurately  recorded  in  the  data 
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system  in  the  last  two  maintenance  cycles  and  one  “missing”  observation  in  the  data  system 
three  maintenance  cycles  ago.  So,  if  during  the  next  maintenance  cycle  the  BCS  was 
accurately  recorded,  the  part  would  be  declared  a  bad  actor  because  there  are  three  recorded 
BCSs  within  a  90-day  period.  The  state  BA  (“Bad  Actor”)  represents  the  detection  and 
elimination  of  a  bad  actor  (an  absorbing  state).  By  defining  the  states  in  such  a  way,  we 
have  implicitly  included  a  90-day  “window”  in  the  model.  The  chain  moves  from  state  to 
state  with  probability  p  (data  system  accuracy)  or  1  -  p  (error  rate). 

Given  this  model,  we  can  determine  the  expected  number  of  maintenance  cycles  that 
a  bad  actor  remains  in  the  maintenance  system  before  it  is  detected  and  eliminated.  To  do 
so,  we  constructed  a  one-step  transition  matrix  (shown  in  Table  A-1)  based  on  Figure  A-1. 
This  matrix  contains  one-step  transition  probabilities  Py,  which  represent  the  probability 

that  the  process  will,  when  in  state  i,  make  a  transition  into  staxej. 


Table  A-1.  Matrix  of  One-Step  Transition  Probabilities  P/y  (where  q  =  1  -  p) 
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Let  X</>  be  the  state  vector  for  which  each  of  its  elements  correspond  to  the 
probability  of  being  in  a  given  state  at  time  i.  Let  X<0>  represent  the  initial  condition  of  the 
system  where  no  observations  have  been  made,  that  is,  X<0>  =  [1,  0,  0,  0,  0,  0,  0, 0]. 
Thus,  the  chain  starts  in  state  MMM. 

To  determine  the  state  probability  vectors  for,  say,  i  =  1,  8,  the  following 

equation  is  used:  X</>  =  X<i  -  1>  ♦  P.  An  interesting  result  obtained  from  this  series  of 
equations  is  the  probability  of  detecting  and  eliminating  a  bad  actor  by  time  /.  The  last 
element  of  the  state  vector  X</>  represents  this  probability,  which  we  will  denote  as 
XBA</>.  Figure  A-2  illustrates  the  probability  that  a  bad  actor  has  been  detected  and 
eliminated  by  time  i  for  a  data  accuracy  of  90  percent 
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Figure  A-2.  Cumulative  Probability  of  Detecting  and  Eliminating  a  Bad  Actor 
by  Time  /  Given  a  Data  Accuracy  of  90  Percent 

The  next  step  is  to  determine  the  probability  distribution  for  detecting  a  bad  actor  at 
time  I.  We  can  compute  this  distribution  from  the  above  cumulative  probabilities.  The 
probability  of  detection  at  time  /  is  the  difference  between  the  cumulative  probability  that  a 
part  is  declared  a  bad  actor  at  time  i  and  the  cumulative  probability  that  it  was  declared  a  bad 
actor  at  time  (/  -  1).  The  distribution  is  shown  in  Figure  A-3. 


Figure  A-3.  Probability  of  Detecting  a  Bad  Actor  at  Time  / 
Given  a  Data  Accuracy  of  90  Percent 


The  expected  number  of  months  that  a  bad  actor  is  in  the  astern  is: 

2  i  ♦  (XBA</>  -  X6A<i  - 1>)  ss  2.377  (i*  *  1, ....  100)  for  a  data  accuracy  of  90  percent 
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This  includes  the  assumption  that  the  first  step  of  the  Markov  chain  was  instantaneous 
(both  a  failure  and  the  opportunity  to  record  the  failure  occurred  at  t  =  0).  Figure  A-4  and 
Table  A-2  represent  the  expected  time  to  detect  bad  actors  for  a  range  of  data  system 
accuracies. 


p 


Dota  Accuracy 

Figure  A*4.  Expected  Lifetime  of  a  Bad  Actor  versus  Data  System  Accuracy 


Tabie  A>2.  Expected  Lifetime  of  a  Bad  Actor 
Given  Leveis  of  Data  Accuracy 


Expected  Lifetime  of 
Actor  (months) 

.50 

7.769 

.60 

5.271 

.70 

3.841 

.75 

3.348 

.80 

2.953 

.90 

2.377 

.98 

2.063 

.99 

2.031 

Thus,  for  a  90>percent  accurate  data  system,  a  bad  actor  should  be  detected,  on  the 
average,  in  less  than  2.5  months. 
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A  second  case  was  investigated.  For  this  case,  an  additional  assumption  was  made: 
At  the  time  of  the  third  failure  (within  a  90-day  period),  the  maintenance  crew  will  first 
check  the  data  system  to  see  the  past  history  of  the  part  and,  if  two  failures  are  found,  then 
accurately  record  the  third  BCS  and  declare  the  part  a  bad  actor.  Thus,  the  probability  of 
accurately  recording  the  third  BCS  that  has  occurred  within  a  90-day  period  is  one.  In  the 
first  case,  there  was  a  positive  probability  that  this  third  occurrence  may  not  be  -ecorded 
accurately  or  at  all.  Also,  the  maintenance  crew  was  not  responsible  for  declaring  a  part  a 
bad  actor. 

Figure  A-5  below  shows  the  Markov  chain  state  transition  diagram  for  the  second 

case. 


The  same  analysis  previously  described  was  done  based  on  the  above  diagram.  The 
following  graphs  represent  the  pertinent  results.  Figure  A-6  represents  the  probability  that  a 
bad  actor  will  be  detected  and  eliminated  by  time  i  for  a  data  accuracy  of  90  percent 


Tlrm  /(Month*) 


Figuro  A-6.  Cumulative  Probability  of  Detecting  and  Eliminating  a  Bad  Actor 
by  Time  /  Given  a  Data  Accuracy  of  go  Percent 
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Figure  A-7  represents  the  probability  that  a  bad  actor  was  detected  at  time  i. 


Figure  A*7.  Probability  of  Detecting  a  Bad  Actor  at  Time  / 

Given  a  Data  Accuracy  of  90  Percent 

Figure  A-8  and  Table  A-3  represent  the  average  amount  of  time  to  detect  a  bad  actor 
for  various  data  accuracies. 


Figure  A-8.  Expected  Lifetime  of  Bad  Aetore  veraus  Data  System  Accuracy 
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Table  A-3.  Expected  Lifetime  of  A  Bad  Actor 
Given  Levels  of  Data  Accuracy 


Expected  Lifetime 

Data  Accuracy  (Months) 

.50 

4.667 

.60 

3.651 

.70 

2.998 

.75 

2.756 

.80 

2.552 

.90 

2.233 

.98 

2.041 

.99 

2.020 

Thus,  for  a  system  with  90  percent  accuracy,  a  bad  actor  should  be  detected,  on  the 
average,  in  approximately  2.2  months.  Comparing  the  two  cases  that  were  considered,  one 
can  see  that  for  lower  levels  of  data  accuracy,  the  average  amount  of  time  to  detect  a  bad 
actor  was  much  higher  in  the  first  case  than  in  the  second  case.  As  data  accuracy  increases, 
the  time  needed  to  detect  a  bad  actor  decreases  at  a  faster  rate  in  the  first  case.  For  higher 
levels  of  data  accuracy,  the  amount  of  time  to  detect  a  bad  actor  approaches  2  months  in 
both  cases.  For  data  accuracy  of  100  percent,  a  bad  actor  can  be  detected  in  2  months. 

Overall,  this  analysis  shows  that  as  levels  of  data  accuracy  increase,  the  amount  of 
time  to  detect  a  bad  actor  decreases.  It  also  provides  some  insight  on  the  range  of  data 
accuracy  that  would  be  sufficient  for  identifying  a  bad  actor.  However,  the  costs  associated 
vkdth  various  levels  of  accuracy  should  also  be  a  factor  in  determining  the  range  of  data 
accuracy. 
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5  0  Manhours  to  support  DIREPS  from  Seymour  Johnson  suppUed  by  DRC 

5 1  List  of  DIREPS  (categorized  by  S/W  Deficiency,  S/W  Mod,  Data  Issue,  Security  Access, 
Printing  Issue,  User  Familiarity,  Functionality  Cbange,Otber)  supplied  by  DRC 
5  2  Report  on  TICARRS  Data  Load  for  Seymour  Johnson  AFB  supplied  by  DRC 
5  3  TICARRS  transaction  rates  during  Seymour  Johnson  test  supplied  by  DRC 
5  4  P016  CDS  Software  Productivity  Analysis 

III.  Cost  Estimates 

A.  CAMS 

1  CAMS  Cost  Benefit  Analysis,  9/30/92 

2  CAMS  USAF-Wide  Funding  Status  Chart,  9/92 

3  Cost  Figures  for  CAMS  from  LGM,  4/22/92 

4  CAMS  USAF-Wide  Funding  Status,  supplied  by  Cliff  Hall,  5/13/92 

5  CAMS  FY93  and  Projected  Manning 

6  CAMS  Prior  Year  Funding 

B.  REMIS 

1  REMIS  Independent  Cost  Estimate  (ICE,  10/92 

2  REMIS  Development  (Tosts  and  Operations  and  Support  Costs,  4/92 

3  REMIS  Life-Cycle  Cost  Estimate,  1 1/8/91 

4  REMIS  Finn-Fixed  Price  Contract  Structure,  3/24/92 

5  REMIS  EcontHnic  Analysis,  prepared  by  Systems  and  Applied  Sciences  Ccwp.,  1 1/27/85 

6  REMIS  Program  Office  Estimate  (POE).  9/92 

7  REMIS  Program  Office  Estimate,  6/91 

8  Cost/Benefils  Analysis  for  REMIS  FY93-FY02. 9/30/92 

9  REMIS  FY1992  Funding  Execution  Status,  5/4/92 

1 0  REMIS  FuncticHial  Economic  Analysis,  9/13/91 

1 1  REMIS  exists  for  Future  Operations  1993-1998,  supplied  by  Liuon 

1 2  Litton  Document  showing  proposal  to  reduce  overall  hardware  and  perscumel  OandM  costs  while 
improving  petfonnance 

1 3  RI^IS  Program  Funding  supplied  by  CHiff  Hall,  5/13/92 

1 4  REMIS  PMO  lYitH'  Yev  Fiuidhig 

1 5  REMIS  Program  Funding  sheets  supplied  by  The  Stratos  Group,  Inc.  on  7/1/93 

1 6  Memorandum  fnan  C!luHdie  Swett,  Litton  Cranputer  Services,  to  Quutes  Crawford,  Subject: 
Pricing  for  Tandem  SAFEGUARD.  8/1 1/93 
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C.  CAMS  and  REMIS 

1  Memorandum  of  AgreemenL  Cost  Documentation  for  CAMS/REMIS,  9/25/92 

2  Memorandum  of  Agreement:  Cost  Documentation  for  CAMS-REMIS,  8/92 

3  CAMS/REMIS  FY93  and  Out  Funding 

D.  TICARRS 

1  Cost  Estimate  for  Activating  the  B-IB  on  TICARRS/SDS,  9/8/92 

2  An  Initial  Business  Case  Analysis  of  TICARRS  ‘92,  12/15/92 

3  DD250s  for  FY87-FY91  Final  Inspection  and  Receiving  Reports 

4  TICARRS  Technical  Cost  Proposal/TICARRS-to-REMIS  Data  Conversion  Task  Order  92-016 

5  Cost  Analysis  to  Field  TICARRS  92  Across  27  AF  Systems  and  Com-Electronics  by  DRC, 
5/25/93 

6  TICARRS  Software  Estimates  vs  Actuals  28  May  1993  by  DRC 

7  Computer  Cost  for  TICARRS  92  Testing  and  CAMS  Enhancements  Increment  VI  and  Beyond, 
from  cuff  Hall.  6/3/93 

8  Document  prepared  by  HaU  organization  Subject:  Costs  of  expanding  TICARRS  scope  and 
functionality 

9  TICARRS  preliminary  write-up  of  cost  to  provide  Maintenance  Production  management 
ctq>ability  supplied  by  DRC 

1 0  Letter  to  Betsy  Bailey  from  Bruce  Cooper  Subject;  Clarification  of  TICARRS  Costs,  5/26/93 

1 1  A  Quote  on  DRC’s  Current  Hardware  Configuration  frcxn  Dataguard  Recovery  Services,  Inc., 
provided  by  DRC,  7/13/93 

IV.  Briefings  and  Presentations 

A.  CAMS 

1  Trip  to  Europe  briefing  by  CUff  Hall  3/1/93 

2  Comparison  of  CAMS  Reporting  and  CEMS  EHrect  Line  Reporting  (DLR)  of  CEMS  Data  Feb. 
‘92-Jan  ‘93 

3  CAMS  Presentation  on  Unit  CAMS  History,  CAMS  UtiUzation,  Individual  Cottcems,  Local 
Improvement  Measures,  Mobile  CAMS  Terminals,  Deployable  Comm  Package 

4  CAMS  Overview  presented  by  John  Hayes 

5  DBM  Concerns  presentation;  Subject:  status  and  iMoblems  of  supporting  CAMS  on  site  and 
proposed  changes  a  la  TICARRS  supplied  by  Gunter  AFB 

6  Presentation  by  Bill  Burson,  Standard  Systems  Center,  Subject:  USAF  ADP  Consolidation 
Program 

B.  REMIS 

1  Operational  Assessment  of  REMIS:  PPS  and  EIMSURS,  briefing  by  detain  John  Robinson, 
12/92 

2  REMIS  Status  Briefing  to  Mr.  Majors,  2/19/92 

3  REMIS  Briefing  to  Mr.  Dave  Rob^,  SubctHiunittee  on  Defense  House  Ap{»o{»iations 
Comm.  7/22/91  by  Ll  Col.  Duane  Johnson,  REMIS  PD 

4  REMIS  Joint  Program  Management  Review,  briefing  by  CUff  Hall  12/19/91 

5  Air  Force  REMIS  Assessment  Repent,  6/27/90 

6  Why  REMIS?  briefing  given  to  Mr.  Rad  by  AFLC,  6/14/90 

7  REMIS  presentation  to  OSD  Action  Officers  by  AFLC,  9/19/90 

C.  CAMS  and  REMIS 

1  Integrated  Wesqptm  System  Managemrat  of  CAMS/REMIS — IWSM  Experience  Report 

2  CAMS/REMIS  briefing  by  Clifford  Hall,  5/13/92 

3  CAMS/REMIS/CEMS:  A  Special  Study,  Iniefing  by  Wayne  Fenwick,  Litttm  CmniHiter 
Services,  2/4/93 

4  CAMS/REMIS  Interface  Design,  briefing  by  Wayite  Fenwidc,  Litton  Cconputer  Services, 
5/26A13 


B-6 


D.  TICARRS 

1  Concepts  of  Operation:  TICARRS  and  SDS  Presentation  by  Clifford  Hall,  8/17/92 

2  Options  for  Implementation  of  Congressional  Language.  12/24/92 

3  Maintenance  Information  System  Architecture  for  the  Air  Force  of  the  Future,  briefing  by  DRC, 
9/92 

4  TICARRS:  The  Solution  To  Two-Level  Maintenance  Information  Requirements  briefing  by 
DRC 

5  nCARRS  Presentation  to  Institute  for  Defense  Analyses  by  Dynamics  Research  Corporation, 
7/28/93 

E.  CAMS,  REMIS,  and  TICARRS 

1  TICARRS  Presentation  to  Cynthia  Kendall  by  DRC,  9/9/92 

2  CAMS,  REMIS.  TICARRS  briefing  by  Clifford  Hall  .9/92 

3  Briefing  by  Cliff  Hall,  “The  Air  Force” 

F .  Other 

1  Coronet  Deuce  Phase  III — Progress  Briefing,  1/21/93-2/3/93 

2  Integrated  Maintenance  Information  System  (IMIS),  briefing  by  Robert  C.  Johnson,  Armstrong 
Labraatory 

3  CASS  AX  Interchange  Meeting  by  Ray  Lebeau,  NSWC/CARDEROCK,  1/15/93 

4  OC-ALC/LAH  B-52/MissiIes  Division,  Reliability  and  Maintainability  Briefing  Presented  by 
Pamela  Herzog 

5  F-22  Advanced  Tactical  Fighter  (ATF)  Integrated  Maintenance  System  AIMS  Briefing  by  Major 
Ralph  Lowry,  F-22  Support  Data  Integrated  Product  Team  Leader,  Wright  Patterson  AFB, 
3/20/93 

6  F-15E  Can-Nol-Duplicate  (CM))  Failure  Reduction  Study  Program 
Completion/Recommendation  Briefing  by  McDonnell  Aircraft  Company,  4/21/92 

7  IMIS:  Integrated  Maintenance  Information  System  briefing  1/26/93  by  GDE  Systons  Inc. 

8  IMIS  Briefing  by  (jeneial  Dynamics  Electronics  Division,  6/4/93 

9  F-22  Advanced  Tactical  Fighter  (ATF)  Integrated  Maintenance  System  AIMS  teiefing  by  Major 
Ralph  Lowry,  Wright  Patterson  AFB,  3/30/93 

V .  Correspondence 

A.  CAMS 

1  Memorandum  For  The  Record  from  Stan  Horowitz,  Sulqect:  The  perspective  on  C/VMS  from 
the  B-1  wing  at  Grand  Forks  AFB,  2/3/93 

2  Memorandum  For  AF/SC,  Sutyect:  Operational  Security  of  Standard  Base  Level  Computers, 
5/14/92 

3  Letter  from  Lockheed  Configuration  Status  Accounting  (CSA)  to  HQ  MSCVSR  and  HQ 
SSC/LGM  Subject  CAMS  TCTO  Rep<«ing.  3/25/93 

B.  REMIS 

1  OSD  CcHisolidated  TandE  Comments  on  REMIS  Test  and  Evaluation  Master  Plan  (TEMP), 
10/26/92 

2  Review  Cmnments  on  REMIS  TEMP  by  David  A.  D(»e,  11/13/92 

3  REMIS  Program  Diiecuv’s  Comments,  3/31/92 

4  Memo  regarding  REMIS  Security  Test  and  Evaluation  (STandE),  2/7/92 

5  REMIS  Functional  Requirements  Board  Meeting  Minutes,  July  16-17, 1991 

6  DCSO/DISDA  Letter  Defense  Data  Netwtuk  (DDN)  x.2S  Qualified  Host  Interfaces,  8/23/91 

7  DISA  (Certificate  of  Tandem  DDN  Access  Nfethod  (DDNAM) 

8  Impact  Statement  on  REMIS  Termination 

9  Memo  to  Dev  Devers  from  Kathleen  Jablonski  Subject:  Action  Item  #6  from  Jai.  1993  IDA 
Visit  to  REMIS  PMO  (Maintenance  Data  Reject  Rate),  4/19/93 

1 0  ACC  message  from  Kathy  Jablonski  Subject:  REMIS  Access  at  Unit  Levd,  4/6/93 

1 1  Mmo  to  M.  Olds  from  D.  Rose;  Sulqect:  Analysis  of  accuracy  rate  for  aircraft  tiansactians  in 
REMIS 
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I  2  List  of  IDA’S  data  requests  to  Cliff  Hall 

1 3  Memorandum  from  Thomas  King,  Litton,  to  Waynard  Devers,  IDA,  Subject:  IDA  Action  Item 
Responses,  7/30/93 

C.  CAMS  and  REMIS 

1  Memo  on  REMIS: . '  ‘  MS/REMIS  Interface  Testing  and  System  Reporting  Designator  (SRD) 
Table  Upload 

2  A  Chronology  c.  User  Concerns  and  Complaints  about  CAMS  and  REMIS,  2/12/93 

3  Impac'.  Statement  on  Proposed  CAMS/REMIS  Termination,  9/23/91 

4  Letter  by  Tom  King;  Subject:  Data  Errors  in  CAMS/REMIS 
Memorandum  for  Chainnan,  Costs/Benefits  Review  Group 

D.  TICARRS 

1  Report  of  Trip  to  DRC  about  TICARRS  92  by  LGMPE,  12/92 

2  Memo  regarding  TICARRS  Demonstration  by  DRC  and  CAMS,  12/17/92 

3  Letter  from  DRC  to  Cliff  Hall,  Subject  Interface  Definition  Support  Requirements,  1/5/93 

4  Letter  from  DRC  to  John  Milligan,  Subject:  Visits  needed  for  info,  on  additional  functionality, 
1/12/93 

5  Memo:  Costing  Proposal  to  Move  TICARRS  to  DDN,  9/1 1/92 

6  Memorandum  from  MSC/SR  to  HQ  USAF,  Subject  TICARRS  Program  Information  Request 
7/24/92  and  TICARRS  Statement  of  Work  FY91-94 

7  Letter  from  I>onald  E.  Desrochers  to  Dev  Devers  5/13/93  Subject  Listing  of  Action  Items 
resulting  from  10-11  May  1993  meetings  at  DRC 

8  Letter  to  MSGT  Randy  Diebold  from  Ray  Pruett,  TICARRS  Field  Operations  5/26/93  Subject 
TICARRS  Contract  F33600-90-D-0360-0004,  Field  Engineer  Support 

9  Message  referring  to  14  April  1993  Letter  from  TICARRS  PMO  concerning  Follow-On 
Contract  5/26/93 

1 0  DRC’s  comments  on  IDA  memos 

I I  Memo  from  DRC  to  Cliff  Hall;  Subject;  Interface  Requirements  for  CEMS  and  SBSS 

1 2  Memo  from  DRC  to  Gilltgan;  Subject:  CEMS  and  SBSS  Interface  Requirements 

1 3  Memo  fen  the  Record  fron  Pat  Kelley  (DRC)  to  Bill  Florae  (IDA)  Subject  Request  for 
TICARRS  Backup/Recovery  Information,  7/12/93 

1  4  Memo  for  the  Reond  from  Pat  Kelley  (DRC)  to  Betsy  Bailey  (IDA)  Subject  Request  for 
TICARRS  Information,  7/12/93 

]  5  Memo  from  John  Nowak,  HQ  USAF/LG,  to  JtAn  Anderegg,  DRC,  Subject  General  Officers 
Working  Group,  7/6/93 

E.  CAMS,  REMIS,  and  TICARRS 

1  Letter  from  DRC  to  Cynthia  Kendall  in  regard  to  9/9/92  presentation,  9/1 1/92 

2  Memorandum  of  Understanding  Between  CAMS/REMIS  and  DRC  regarding  TICARRS 
12/17/92 

3  ACC  PositicR]  Paper,  Subject  Switching  from  CAMS/REMIS  to  TICARRS  ‘92,  12/4/92 

4  A  ReiqxHise  to  the  Suggestion  that  CAMS  should  be  bypassed  and  data  inputted  directly  into 
TICARRS  by  Tom  King,  Litton  Computer  Services 

5  Memorandum  of  Agreement  Reconciling  TICARRS  and  CAMS/REMIS  Functionality 

6  Letter  from  Barry  Phillips,  Genesis  Software  Applications  to  IDA  5/27/93  Subject  Funding 
data  for  CAMS/REMIS/TICARRS 

F.  Other 

1  “New  Approach  Identifies  Malidous  System  Activity”  Major  Patrick  Phillips,  USAF,  Signal, 
3/92 

2  Equipment  Maintenance  Database  PAR-LOG-LCR.-06S-333, 8/84 

3  D^  Automation  Requirement  for  Contractual  Service,  9/83 

4  Statement  (rfWwk  for  the  Development  of  Integrated  Maintenance  Management  System 
(IMMS)  1984 

5  Leoer  from  Taa^lem  Computers  Incorporated;  Sidrject:  Tandon  System  National  Computer 
Security  Canter  C2  Evalumion,  4/7/92 


B-8 


6  Draft  “Leading  Edge”  Article:  Technology  Insertion 

7  SSC/AQ  Request  for  Waiver  to  Allow  Use  of  Group  User- ID/Pass  words  1/23/90  and  HQ 
USAF/SCT  Response  Letter.  4/6/90 

8  CAMS/REM IS-TICARRS  Coasiderations  and  Estimated  Financial  Implications  from  David  A. 
Dore  OASD.  9/14/92 

9  AF  wasted  millions  on  computer  technology,  report  says  by  Jim  Abrams,  AF  Times,  1/4/93 

1 0  Memo  from  ASC/YF  to  IDA,  Subject:  Information  Required  by  IDA  to  Complete  Study  “A 
Comparison  of  CAMS/REMIS  and  TICARRS”  (AIMS  Development  Costs),  6/1 1/93 

1 1  Memorandum  for  ASD(C3I),  Subject:  Comments  on  Draft  IDA  Comparison  of  CAMS/REMIS 
and  TICARRS.  7/30/93 

1 2  Official  OSD  Comments  on  IDA  Paper  P-2863,  “A  Comparison  of  /Sir  Force  Data  Systems" 
frcwn  Robert  T.  Mason,  Assistant  Deputy  Under  Secretary  (Maintenance  Policy) 

1 3  Memorandum  for  Director,  Maintenance  Policy,  ODUSD  (Logistics)  Subj;  IDA  Review 
Versicm  Report  “A  Ccxnparison  of  Air  Force  Data  Systems,”  7/21/93 

1 4  Memorandum  for  Director,  Maintenance  Policy,  ODUSD  (Logistics)  from  David  Dore,  Subjert: 
IDA  Review  Version  Report  “A  Compariscm  of  Air  Force  Data  Systems,"  7/30/93 

1 5  Memorandum  for  Director,  Maintenance  Policy  from  Francis  L.  McDonald,  Staff  Analyst,  Force 
Structure  and  Infrastructure  Cost  Analysis  Division  Subject:  IDA  Report  on  CAMS/REMIS 
and  TICARRS.  7/26/93 

1 6  Comments  on  IDA  Paper  P-2863  “A  Comparison  of  Air  Force  Data  System.s”  by  Litton 
Computer  Services  (Final  Version),  7/26/93 

1  7  Comments  on  IDA  Paper  P-2863  “A  Comparison  of  Air  Force  Data  Systems"  by  Litton 
Computer  Services  (D^t  Version),  7/23/93 

1 8  Comment  on  IDA  Report  “A  Comparison  of  Air  Fwce  Data  Systems”  by  Dynamics  Research 
Corporation,  7/23/93 

VI.  System  Reports  and  Screens 

A.  CAMS 

1  CAMS  data  base  schema  and  associated  stats,  supplied  by  Gunter  AFB 

2  CAMS  Menu  and  slides  showing  CAMS  WCE  processing  supplied  by  Gunter  AFB 

3  Copies  of  LOC  for  CAMS  software  supplied  by  Gunter  AFB 

4  CAMS  Main  Menu  Directory 

B.  REMIS 

1  Copies  of  REMIS-EIMSURS  screens,  1/26/93 

2  Copies  of  REMIS-GCSAS  screens,  11/10/89 

3  Copies  of  REMIS-PPS  screens,  1/25/93 

4  Examples  of  REMIS  Reports:  AV/Trainer  Status/Utilization  Report,  Assignment/Possession 
Report 

C.  TICARRS 

1  TICARRS  PLP  Samples 

2  TICARRS  Canned  (Juery  Samples 

3  Description/Purpose  of  TICARRS  saeens 

4  Examples  of  TICARRS  Data  Outputs  from  Phil  Hermes 

5  Data  Sheets  describing  TICARRS  statistics,  i.e.  database  size,  transaction  rates,  computer  type, 
back  up  procedures,  COTS  software  elements  (supi^ied  by  DRC) 

6  TICA^S  data  base  diagrams  ^d  schema  listing  supplied  by  DRC 

7  Number  of  screens  by  week  (and  daily  averages)  for  Seymour  Johnson  siqiplied  by  DRC 

VII.  House  Appropriations  Committee  Report  Language  for  CAMS/REMIS  and 

TICARRS 

A .  FY93  HAC  Report  Language 

I  Impaa  Information  Regarding  FY93  HAC  Report  Language  on  CAMS/REMIS  and  TICARRS 
9/9/92.  9/4/92 
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2  Memos  from  Kathleen  Jabionski  to  Dave  Dore,  Subject;  Your  Questions  to  FY93  HAC 
Language,  9/11/92 

3  DRC’s  Response  to  FY93  HAC  Report  Language  on  CAMS,  REMIS,  and  TICARRS.  9/14/92 

4  Memo  from  Kathleen  Jabionski  to  Dave  Dore  regarding  HAC  FY93  Language,  8/14/92 

5  REMIS  and  TICARRS  Direction  8/10/92  from  Robert  Heath  (Robbins-Gioia)  to  MSC/SR 

6  SAC  Defense  Subcommittee  Results,  9/16/92 

7  GAO  Correction  Plan  9/4/92 

8  Responses  to  OSD  Questions  from  MSC/SR,  9/9/92 

9  DASD(IS)  Memo,  7/31/92 

I  0  DoD  Appropriations  Bill,  1993  p.38-41,  238-239 

I I  DoD  Appeal  FY93  Defense  Appropriations  Bill,  9/4/92 

1  2  Cynthia  Kendall's  Questions  and  3  Sets  of  Replies 

1 3  DRC’s  Response  to  SPO  “Impact  Information  Regarding  FY93  HAC  Report  Language  on 
CAMS/REMIS  and  TICARRS.  8/18/92 

1 4  Response  to  OSD  Questions  by  SPO  “Impact  Information...’’,  9/9/92 

1 5  DRC’s  Response  to  SPO’s  Response 

1 6  DRC  Response  to  OSD  Questions  Regarding  TICARRS,  8/17/92 

1 7  DRC  Response  to  OSD  QuesUon  #3— TICARRS  Funding  Profile  FY87-FY91 

1 8  1992  Robbins-Gioia,  Inc.  Documentation  for  the  Study  of  CAMS/REMIS/TICARRS 

1 9  DoD  Concerns  about  FY93  House  Comm,  on  Appropriations  Report  Language  on 
CAMS/REMIS 

2  0  TICARRS  CBD  Notice  for  FY93.  9/1 1/92 
B.  FY92  HAC  Report  Language 

1  CAMS/REMIS  Concerns  in  DoD  FY92  Appropriations  Conference  Report 

2  FY92  Appropriations  Bill  Conference  Report  102-328, 9/18/91 

3  Letter  from  GAO/IMTEC  regarding  House  Report  102-328,  FY92  Appropriations  Bill 
Conference  Report 

4  FY92  Appropriations  Bill  SAC  Report 

VIII.  Miscellaneous  Reports 

1  Maintenance  Data  Collection  Review  (On-Equipment  Failure  Data)  by  Air  Force  Logistics 
Management  Center,  Gunter  AFB,  10/91 

2  Depot  Maintenance  Management  Information  System  Benefit  Analysis  and  Combat  Capability 
Assessment:  Summary  and  Conclusions  by  Synergy,  Inc.,  S/25/88 

3  52nd  Fighter  Wing:  Aircraft  Maintenance  Summary  12/92 

4  Deployable  Log  C2  Requirements  Capability  Technical  Repon  by  Booz-AUen  and  Hamilton, 
Inc.,  1/92 

5  Concept  of  Future:  Air  Fence  Logistics  Maintenance  Information  Systems — ^Draft,  9/25/92 

6  Maintenance  Data  Collection  Documents  for  EDA  from  OC-ALC/FMl,  3/9/93 

7  Depot  Maintenance  Automated  Data  Systems  by  Bnancial  Management  Directorate  Wamo- 
Robins  ALC 

8  VFL  Recommendaiions/Comments  on  McAir  Silver  Bullet  Study 

9  RandM  2000  Field  Data  Requirements  for  a  SPO  Operation  by  Phillip  Hermes  ASD/ENSSR 
NAECON  92  Paper.  Dayton,  OH.  5/21/92, 

1 0  NAECON  92  Information  Systems  in  the  User  Environments  RandM  2000  Field  Data 
Requirements  for  a  SPO  Operation  Briefing  21  May  92  by  Phillip  Hermes  ASD/ENSSR 
WPAFB 

1 1  Hermes’  Perspectives  on  Field  Data  Issues  Jan.  93  by  Phillip  Hermes  ASCVENSSC  WPAFB 

1 2  FDTS  Field  Data  Tracking  Systems  for  Program  Offices  (AFSC  and  AFLC)  by  Phillip  Hermes 
1984  RandM  Workshop  McClellan  AFB 

1 3  Total  Quality  Management  and  Field  Data  Systems  by  Phillip  Hermes 

1 4  The  Quality  Flyer  Special  Edition  "Survey  Says...’’,  3/93 

1 5  RqxMls  cm  Squadron  Centered  Logistical  Systems  supplied  by  DRC 
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1  6  GAO  Report;  Air  Force  ADP  Lax  Contract  Oversight  Let  to  Waste  and  Reduced  Competition, 
11/92 

I  7  IMIS  Initital  Estimates  of  System-Wide  Co.sis  and  Benefits:  An  Executive  Summary  by  Dr. 

Burke  Burright,  Capt.  Bradley  Lloyd  (Armstrong  Laboratory),  and  John  Zawiia,  Robbins-Gioia 
Inc.,  5/93 

IX.  Memos  for  the  Record  from  IDA  Study  Team 

1  Memo  for  the  Record  from  Karen  Tyson.  5/13/93.  Subject:  Visit  with  Air  Vehicle  Data 
Officers-REMIS  Users  on  3/30A93 

2  Memo  for  the  Record  from  Karen  Tyson,  5/6/93,  Subject:  Trip  Report  for  Visit  to  SPO’s  and 
IMIS  Lab  on  March  29  and  30,  1993 

3  Memo  for  the  Record  from  Lee  Dymond,  5/6/93,  Subject:  Modificatifflis  to  TICARRS  needed  to 
give  it  the  functions  of  CAMS 

4  Memo  for  the  Record  from  Lee  Dymond.  4/7/93,  Subject  DRC  data  on  F-16s  and  F-15s 

5  Memo  for  the  Record  from  Lee  Dymond.  4/8/93,  Subjea;  Observations  of  Work  Centers  at  S-J 
AFB  Pre-  and  Post-TICARRS — Plans  and  Scheduling 

6  Memo  for  the  Record  from  Lee  Dymond.  4/8/93,  Subject;  Observations  of  Work  Centers  at  S-J 
AFB  Pre-  and  Post-TICARRS — Engine  Management 

7  Memo  for  the  Record  from  Lee  Dymond,  4/8/93,  Subject:  Observations  of  Work  Centers  at  S-J 
AFB  Pre-  and  Post-TICi\RRS — Maintenance  Instruction 

8  Memo  for  the  Recotd  from  Stan  Horowitz,  5/4/93,  Subject:  Modifications  to  TICARRS  needed 
to  give  it  the  functions  of  CAMS 

9  Memo  for  the  Record  from  Lee  Dymond,  5/4/93,  Subject:  Modifications  to  TICARRS  needed  to 
give  it  the  functions  of  CAMS 

1 0  Memo  for  the  Record  from  W.  C.  Devers,  5/5/93,  Subject:  Modifications  to  TICARRS  needed 
to  give  it  the  functions  of  C/^S 

I I  Memo  for  the  Record  from  Lee  Dymond.  5/5/93,  Subject:  Modifications  to  TICARRS  needed  to 
give  it  the  functions  of  CAMS 

1 2  Memo  for  the  Record  from  Stan  Horowitz.  5/5/93,  Subject:  Modifications  to  TICARRS  needed 
to  give  it  the  functions  of  CAMS 

1 3  Memo  for  the  Record  from  Lee  Dymond,  5/5/93,  Subject;  Modificaticms  to  TICARRS  needed  to 
give  it  the  functions  of  CAMS 

1 4  Interoffice  Memo  from  Lee  Dymond,  6/2393,  Subject  SBLC  costs  provided  by  Mr.  Jesse  B. 
Smith  HQ  ACC/SCMD 

1 5  Memorandum  for  the  Record  from  Stan  Horowitz,  IDA,  Subject:  Possible  schedule  for  the 
introduction  of  TICARRS  «i  a  fleet-wide  basis,  7/28/93 
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APPENDIX  C 

QUESTIONNAIRES  ADMINISTERED  DURING  THE 
OPERATIONAL  ASSESSMENT  AT  SEYMOUR  JOHNSON  AFB 


1993  AIR  FORCE  CAMS  USER  QUESTIONNAIRE 


PURPOSE  OF  QUESTIONNAIRE 

The  Air  Force,  through  the  Office  of  the  Secretary  of  Defense,  has  asked  the  Institute 
for  Defense  Analyses  (IDA)  to  conduct  an  independent  analysis  of  two  major  Air  Force 
information  systems — CAMS/REMIS,  and  TICARRS.  IDA  needs  to  understand  the  ways 
pec^le  use  the  information  systems  in  their  jobs  and  their  reactions  to  the  systems.  The 
assessment  that  is  beginning  at  4th  Wing  provides  a  unique  opportunity  to  ^  this.  As  a 
user,  you  can  provide  us  with  valuable  information  about  your  experiences  with  the 
systems.  This  brief  questionnaire,  which  should  take  less  than  10  minutes  to  complete. 
aste  you  for  this  information  about  CAMS. 

Your  name  and  phone  number  would  albw  us  to  reach  you  if  we  have  questions 
about  your  response.  However,  this  is  entirely  optional.  Only  aggregated  responses,  rK>t 
individual  ones,  will  be  reported. 

Work  Center _ Work  Center  Code _ Rank _ AFSC: _ 


Shift  (circle):  Day  Mid  Swing  Name  (optional) _ Phone  (optional). 


INSTRUCTIONS  FOR  COMPLETING  THE  QUESTIONNAIRE 


•  Make  heavy  black  marks  that  fill  the  circle  for  your  answer. 

•  Unless  otherwise  specified  in  the  instructions  for  a  question,  only  PDfi  answer  should  be 
marked. 
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BACKGROUND  INFORMATION 


IN  THIS  SECTION.  YOU  WILL  BE  ASKED  QUESTIONS  ABOUT  YOUR 
BACKGROUND  AND  PAST  INFORMATION  SYSTEM  USE. 

1 .  What  is  your  current  paygrade? 

O  E-1  to  E-3  O  0-1  to  0-3 

OE-4toE-5  0  0-4  to  0-6 

O  E-6  to  E-7  O  0-7  and  above 

O  E-8  and  E-9 

2.  If  you  perform  any  of  the  following  functions,  please  indicate: 

O  Debrief 

O  Flight  line  crew  chief 
O  Plans  and  scheduling 
O  Right  tine  engine  maintenance 
O  Database  management 
O  Analysis 

3 .  What  is  the  highest  school  grade  or  academic  degree  that  you  have 
completed? 

O  Less  than  12  years  of  school  (no  diploma) 

O  GED  or  other  high  school  equivalency  certificate 
O  High  school  diploma 
O  Some  college,  but  did  not  graduate 
O  2-year  college  degree  (AA/AS) 

O  4-year  college  degree  (BA/BS) 

O  Some  graduate  school,  but  no  post-graduate  degree 
O  Post-graduate  degree 

4.  How  long  have  you  been  using  computers  (any  type~mainframes,  PCs.  word 
processors,  etc.)? 

O  6  months  or  less 

O  Between  6  months  and  2  years 

O  Over  2  years 

5 .  Do  you  have  a  personal  computer  at  home? 

OYes 

ONo 


OWG-1toWG-6 
OWG-7toWG-11 
O  WG-12and  above 
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INFORMATION  SYSTEMS  EXPERIENCE 


1 

6.  How  satisfied  are  you  with  the  following  aspects  of  CAMS? 

Mark  only 

cne 

1 

answer  for  each  item. 

0am 

■ 

V«y 

Sittid 

Danll&iaif 

■ 

Ease  of  use,  understandability 

0 

o 

0 

o 

o 

0 

1 

Ability  to  input  data  easily 

o 

o 

o 

0 

0 

0 

■ 

Ability  to  get  data  out  easily 

0 

0 

0 

0 

0 

0 

1 

Ability  to  modify  previously  input  data 

o 

0 

0 

0 

0 

o 

Usefulness  of  training 

0 

0 

0 

0 

o 

0 

1 

Usefulness  of  manuals,  documentation 

0 

o 

0 

0 

0 

0 

Ability  to  help  me  perform  my  duties 

0 

o 

0 

0 

o 

0 

1 

Performance  of  functions  1  really  need 

0 

0 

o 

o 

o 

0 

■ 

Perfoimance  of  functions  that  are  optional  for  me 

o 

o 

o 

0 

0 

0 

1 

Availability  when  1  need  it 

0 

0 

0 

0 

o 

0 

1 

Ability  to  input  data  quickly 

0 

o 

0 

o 

0 

0 

■ 

Ability  to  get  data  out  quickly 

0 

0 

0 

0 

o 

0 

1 

Usefulness  of  reports  generated  from  CAMS 

0 

0 

0 

0 

0 

0 

Accuracy  of  data  obtained 

0 

0 

0 

o 

o 

o 

Overall  satisfaction  wKh  CAMS 

0 

0 

0 

0 

0 

0 

Dm 

Uwtll 

1-6 

SitnOia 

1-3 

Monttan 

""•aav 

1 

1  mortH 

aalta 

JjCHI 

ma 

1 

Length  of  time  you  have  used  CAMS 

0 

o 

0 

0 

0 

0 

AtMtIknM 

AtowBrr— 

Alflliiw 

Obw 

■ 

QW 

tumH 

AMI 

1 

How  often  do  you  use  CAMS 

0 

o 

o 

0 

0 

0 

Oa 

Jm 

Sot 

faut 

iSMKiOMt 

1 

Hours  spent  using  CAMS  during  a  normal  workday  O 

o 

0 

0 

O 

0 

Ar«  you  a  hands-on  usar  of  CAMS  (that  is,  you  uss  the  terminal  yourself  to 
input  data  or  get  data  out)?  Yes  No 


Are  there  particular  features  of  CAMS  that  make  It  easy  to  use? 

Yes  No 


I  "  - . . 

I 
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Have  you  had  any  problems  using  CAMS?  Yes  No 

if  yes,  what  kinds  of  problems?  Be  specific. 


Do  you  use  CAMS  reports  in  your  job?  Yes  No 

if  yes,  which  reports? 


Do  you  ever  find  it  necessary  to 
if  yes,  give  details. 


verify  reports  against  another  source? 

Yes  No 


7.  Have  you  ever  used  TiCARRS  or  F-16  CDS  or  F-117  Smart  Data  System? 
if  yes,  how  satisfied  were  you  with  the  system? 

Dm 

V«y  mcM  Vir  noliptiy 

ftUrttd  KMM  QlBMIlAd  QlMAiBid  DmUKmi 

Overall  satisfaction  0  0  0  0  0  0 

8.  Does  your  Job  generally  involve  inputting  data,  outputting  data,  or  both? 

O  Inputting  data  O  Both  inputting  and  outputting  data 

O  Outputting  data  O  Using  data  obtained  from  someone  else 

9.  in  general,  which  of  these  statements  comes  CLOSEST  to  describing  how 
important  CAMS  is  to  performing  your  Job?  Mark  only  one  answer. 

O I  could  not  do  my  job  without  the  system. 

O  Without  the  system,  I  could  do  my  job,  but  it  would  be  extremely  difficuH. 

O  Without  the  system,  I  could  do  my  job,  with  some  additionai  time  or  cost  required. 

O  The  system  is  nice  to  have,  but  it  is  not  critical  to  my  job. 

O I  could  do  my  job  more  easily  without  the  ^stem. 


10.  If  you  have  comments  about  CAMS  or  TICARRS  please  write  them  here. 


WEEK  6  TICARRS  ASSESSMENT  QUESTIONNAIRE 


PURPOSE  OF  QUESTIONNAIRE 

The  Air  Force  and  the  Institute  for  Defense  Analyses  (IDA)  need  your  help  in  the  assessment 
of  TICARRS  at  4th  Wing.  As  a  user,  you  can  provide  us  with  valuable  information  about  your 
experiences  with  TICARRS.  This  questionnaire  ^ould  take  less  than  10  minutes  to  complete. 

Your  name  and  phone  number  would  allow  us  to  reach  you  if  we  have  questions  about  your 
response.  However,  this  is  entirely  optional.  Only  aggregated  responses,  not  individual  ones,  wll 
be  reported. 

Work  Center _ Work  Center  Code _ Rank _ 

AFSC: _ Today's  Date _ 

Shift  (circle):  Day  Mid  Swing  Name(optionaO _ Phone(optiona!) _ 


INSTRUCTIONS  FOR  COMPLETING  THE  QUESTIONNAIRE 


•  Make  heavy  black  marks  that  fill  the  circle  for  your  answer. 

•  •  Unless  otherwise  ^}eciried  in  the  instructions  for  a  question,  mark  only  fiofi  answer. 

•  For  Yes/No  questions,  please  circle  the  appropriate  response. 

•  Some  questions  ask  for  a  written  reponse  rather  than  blackening  a  circle.  In  such  cases, 
write  responses  in  the  spaces  provided. 


05 


BACKGROUND  INFORMATION 


1 .  What  is  your  currant  paygrade? 

O  E-1  to  E-3  O  0-1  to  0-3  O  WG-1  to  WG-6 

O  E-4  to  E-5  O  0-4  to  0-6  O  WG-7  to  WG-1 1 

O  E-6  to  E-7  O  0-7  and  above  O  WG-1 2  and  above 

O  E-8  and  E-9 

2.  If  you  perform  any  of  the  following  functions,  please  indicate: 

O  Debrief 

O  Right  line  crew  chief 
O  Plans  and  scheduling 
O  Right  line  engine  maintenance 
O  Database  management 
O  Analysis 

3 .  What  is  the  highest  school  grade  or  academic  degree  that  you  have 
completed? 

O  Less  than  12  years  of  school  (no  dipbma) 

O  GED  or  other  high  school  equivalency  certificate 
O  High  school  diploma 
O  Some  college,  but  did  not  graduate 
O  2-year  college  degree  (AA/AS) 

O  4-year  college  degree  (BA/BS) 

O  Some  graduate  school,  but  no  post-graduate  degree 
O  Post-graduate  degree 

4.  How  long  have  you  been  using  computers  (any  type-mainframes,  PCs,  word 
processors,  etc.)? 

O  6  months  or  less  O  Between  6  montiis  and  2  years  O  Over  2  years 

5.  Do  you  have  a  personal  computer  at  home? 

OYes  ONo 
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S/.TIS FACTION  WITH  TICARRS 


6 .  How  satisfied  are  you  with  the  following  aspects  of  TICARRS? 
answer  for  each  item. 


Ease  of  use,  understandability 
Ability  to  input  data  easily 


Mark  only  one 


O 

O 


Dm* 

Vwy  nolapfiy 

UBHBHIB  UDDUUBK 

o  o 

o  o 


Ability  to  get  data  out  easily 
Ability  to  modify  previously  input  data 


O  O  O  O  O 

O  O  O  O  O 


O 

O 


Usefulness  of  training 

Usefulness  of  manuals,  documentation 


0  0  0 

O  O  O 


O  O  O 

O  O  O 


Ability  to  help  me  perform  my  duties 
Performance  of  functions  I  really  need 


O  O  O  O  O 

O  O  O  O  O 


O 

O 


PerformarK:e  of  functiorts  that  are  optional  for  me  O  O  O 

Availability  when  I  need  it  0  0  0 


O  O  O 

O  O  O 


Ability  to  input  data  quickly 
Ability  to  get  data  out  quickly 


O 

O 


O  O  O  O  O 

0  0  0  0  0 


Usefulness  of  reports  generated  from  TICARRS  O  O 

Accuracy  of  data  obtained  O  O 


O 

O 


0  0  0 
0  0  0 


Overall  satisfaction  with  TICARRS 


Vwy 


o  o  o  o  o  o 


iMmftm 

Length  of  time  you  have  used  TICARRS  O 


Mowllan 

How  often  you  use  TICARRS  O 

UMf  Qw^ 

Hours  using  TICARRS  during  a  normal  workday  O 


Mm  than 


nolafliy 

OatflKiBif 


o  o 


QW 

O 

Oat 

O 


AMwIlm—  AttwUmn  Atawllmn 


O  O  O  o 


o 


THf— 


O 


Eac 

o 


o 
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Are  you  a  hands-on  user  of  TICARRS  (that  is,  you  use  the  terminal  yourself 


to  input  data  or  to  get  data  out)? 

Yes 

No 

Are  there  particular  features  of  TICARRS 
that  make  it  easy  to  use? 

Yes 

No 

If  yes,  please  provide  details  on  these  features: 


Have  you  had  any  problems  using  TICARRS?  Yes  No 

If  yes,  what  kinds  of  problems?  Be  specific. 


Do  you  use  TICARRS  reports  in  your  job?  Yes  No 

if  yes,  which  reports? 


Do  you  ever  find  it  necessary  to  verify  reports 

against  another  source?  Yes  No 

If  yes,  give  details. 


7.  How  satisfied  were  you  with  CAMS  (the  maintenance  data  system  used  by 
4th  Wing  before  this  assessment)? 


Vt<y  MbcwT 

SHmm  BmUllltd  Nmihmr 

o  o 


Dem 


Overall  satisfaction  with  CAMS 


O 


O 


O 


O 


Plaasa  compara  how  satisfiad  you  ara  with  TICARRS  relative  to  CAMS. 
For  each  aspect,  consider  whether  TICARRS  is  better  or  worse  than  CAMS^ 


Ease  of  use,  understandabil'ity 

Ability  to  input  data  easily 

Mu<hb«nar 

ttanCAMS 

O 

0 

Somn*\tt 

Iwnef 

0 

0 

II  oo 

SonwmhM 

than  CAMS 

0 

0 

MuchWMM 

imnCAMS 

O 

0 

Don 

notip^ 

O 

0 

Ability  to  get  data  out  easily 

o 

o 

o 

o 

0 

o 

Ability  to  modify  previously  input  data 

o 

0 

0 

0 

0 

0 

liiiliiiii 

iiiliii 

Usefulness  of  training 

o 

0 

0 

0 

6 

o 

Usefulness  of  manuals,  documentation 

0 

0 

o 

0 

0 

0 

Ability  to  help  me  perform  my  duties 

0 

o 

0 

0 

0 

0 

Performance  of  functions  1  really  need 

0 

0 

0 

0 

0 

0 

Performance  of  functions 

that  are  optbnal  for  me 

0 

0 

o 

0 

0 

0 

Availability  when  1  need  it 

0 

o 

o 

o 

0 

Ability  to  input  data  quickly 

0 

o 

o 

6 

O 

b 

Ability  to  get  data  out  quickly 

o 

0 

0 

0 

0 

o 

Usefulness  of  reports 

generated  from  TICARRS 

0 

o 

0 

0 

0 

0 

Accuracy  of  data  obtained 

0 

0 

0  0  0  0 

Z>'> 

1 

Overall  satis^ction  with  TICARRS 

relative  to  CAMS 

0 

o 

0 

O 

0 

0 

8.  Does  your  job  generally  involve  inputting  data,  outputting  data,  or  both? 

O  Inputting  data  O  Both  inputting  and  outputting  data 

O  Outputting  data  O  Using  data  obtained  from  someone  else 

9.  in  general,  which  of  these  statements  comes  CLOSEST  to  describing  how 
important  TiCARRS  is  to  performing  your  Job?  Mark  only  one  answer. 

O I  could  not  do  my  job  without  the  system. 

O  Without  the  system,  I  could  do  my  job,  but  it  would  be  extremely  difficult. 

O  Without  the  system.  I  could  do  my  job,  vi^h  some  additional  time  or  cost  required. 
OThe  system  is  nice  to  have,  but  it »  not  critical  to  my  job. 

O I  could  do  my  job  more  easily  without  the  system. 

10.  If  you  have  comments  about  CAMS  or  TICARRS  please  write  them  here. 
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APPENDIX  D 


IDA  TEST  OF  REMIS  FUNCTIONALITY  AND 
SELECTED  OPERATING  CHARACTERISTICS 


APPENDIX  D 


IDA  TEST  OF  REMIS  FUNCTIONALITY  AND  SELECTED 
OPERATING  CHARACTERISTICS 


BACKGROUND 

This  appendix  describes  the  results  of  an  IDA  test  of  REMIS  functionality  and 
status  performed  in  late  May  1993.  This  test  supported  several  pans  of  our  report, 
including  the  sections  in  Chapter  V  on  functionality  and  operating  characteristics,  and 
Chapter  Vn  on  costs.  There  were  four  major  reasons  for  this  detailed  review  of  REMIS: 

(1)  There  has  been  congressional  concern  (generated  at  least  in  part  by  General 
Accounting  Office  oversight  studies)  about  the  progress  and  ultimate  operation 
of  REMIS.  These  concerns  are  partly  responsible  for  this  study. 

(2)  Through  visits  to  several  bases,  two  depots,  and  one  Major  Command 
(MAJCOM),  we  were  able  to  observe  CAMS,  REMIS,  and  TICARRS  in 
operation  early  in  the  study.  Because  there  are  few  REMIS  users  (relative  to 
projections  of  use  after  full  operational  capability  is  reached),  we  had  fewer 
opportunities  to  observe  its  use  than  we  had  to  observe  the  use  of  CAMS  and 
TICARRS.  Although  formal  operational  test  and  evaluation  was  conducted  for 
the  Product  Performance  Subsystem  (PPS)  and  the  Equipment  Inventory, 
Multiple  Status  and  Utilization  Repotting  Subsystem  (EIMSURS)  of  REMIS, 
the  Air  Force  has  not  yet  issued  its  report,  so  we  could  not  use  that  as  a  basis 
for  our  analysis.  Formal  demonstrations  are  not  a  complete  substitute  for 
observing  a  system  in  actual  use. 

(3)  To  support  our  inquiry  into  the  impact  of  information  systems  on  weapon 
system  logistics  measures  and  mission  capability,  we  requested  historical  data 
from  both  the  REMIS  and  TICARRS  Program  Management  OfEces  (PMOs). 
The  TICARRS  staff  provided  the  requested  data  within  about  two  weeks;  but 
the  REMIS  staff  could  not  completely  fulfill  the  request,  and  we  had  to  look  to 
other  sources  (e.g.,  hard-copy  data  from  the  Air  Combat  Command). 

(4)  At  the  time  of  the  test,  M  AISRC  HI  for  REMIS  was  scheduled  for  September 
1993.  IDA’S  limited  observations  of  REMIS  in  actual  use  raised  questions 
about  the  reasonableness  of  that  schedule.  (In  fact,  the  schedule  has  since  been 
pushed  back.)  REMIS’s  status  and  schedule  had  to  be  determined  in  order  for 
IDA  to  make  realistic  cost  estimates. 
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For  alJ  these  reasons,  IDA  decided  to  make  a  more  detailed  assessment  of  the 
current  functionality  and  status  of  REMIS  and  to  compare  TlCARRS’s  and  REMIS’s  data 
output  capabilities.  We  gave  the  CAMS/REMIS  PMO  and  the  F- 15  System  Program  Office 
(SPO)  the  same  two  data  requests  in  late  May  1993.  The  first  data  request  was  submitted 
three  days  before  our  arrival;  the  second  was  given  upon  our  arrival  at  the  respective 
Dayton,  Ohio,  offices.  The  requests  (shown  in  Figures  D-1  and  D-2)  covered  several 
aircraft  types,  and  the  data  requested  are  typical  of  those  we  had  seen  requested  by  base. 
Air  Logistics  Center  (ALC),  and  MAJCOM  users.  The  test  focused  on  both;  (1)  the  basic 
functions  of  REMIS  and  (2)  specific  areas  of  functionality  that  TICARRS  users  have 
requested  be  provided  before  TICARRS  would  be  fully  replaced  by  CAMS/REMIS. 

The  responses  to  the  IDA  data  requests  are  summarized  in  Tables  D-1,  D-2,  and 
D-3.  “No  data  found”  means  that  we  were  provided  with  a  record  of  a  query,  but  the 
system  had  no  data.  “Not  provided”  means  that  the  response  did  not  include  a  queiy  for 
that  ite;  “No  capability”  means  that  the  system  does  not  have  the  capability  to  perform  the 
query. 

SEYMOUR  JOHNSON  DATA— GENERAL  CAPABILITY  OF  THE 
SYSTEMS 

Both  REMIS  and  TICARRS  were  able  to  provide  basic  statistics — mission-capable 
rates  and  possessed  hours — for  the  F-15E  aircraft,  4th  Fighter  Wing,  at  Seymour  Johnson 
AFB.  Possessed  hours  agreed  within  1  percent  in  the  two  data  systems,  while  mission 
capable  rates  agreed  within  0. 1  percentage  point  (see  Table  D-3).  However,  REMIS  found 
no  data  for  March  and  April  of  1993. 

The  reason  for  this  lack  of  data  is  relevant  to  the  technical  and  schedule  status  of 
CAMS/REMIS.  In  December  1992,  the  CAMS/REMIS  PMO  began  implementing  an 
initiative  to  allow  for  hourly  updates  to  EIMSURS  through  the  Defense  Data  Network 
(DDN),  rather  than  the  daily  updates  available  through  Autodin.  Initial  implementation  of 
the  initiative — dubbed  Recovered  Functionality  (RECFU)  n  because  it  recovered  planned 
functionality  that  had  to  be  delayed  due  to  funding  cuts — precipitated  a  host  of  problems  for 
REMIS  users.  REMIS  functions  that  had  been  working  fine  stopped  working.  We  received 
reports  of  problems  from  users  at  Hill  AFB,  Tinker  AFB,  Lockheed,  and  from  a  group  of 
MAJCOM  Aerospace  Vehicle  Distribution  Officers.  When  we  visited  Oklahoma  City  ALC, 
REMIS  users  there  were  unable  to  demonstrate  important  functions  to  us,  because  of 
RECFU  II.  The  effort  to  recover  from  RECFU  11  took  six  months.  It  involved  removing 
blocks  of  historical  data  from  the  EIMSURS  module  of  REMIS  and  replacing  it  with  data 
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from  CAMS,  to  ensure  that  CAMS  and  REMIS  had  the  same  data.  Thus,  cenain  historical 
data  were  unavailable  at  times,  including  some  of  the  items  we  requested. 

REMIS  Data  Request — 21  May  1993 

1.  Monthly,  October  1992  through  April  1993,  for  F-ISE  aircraft.  4th  Wing: 

A.  Mission-capable  rates  number  and  percent  of  aircraft  FMC,  PMC,  and  NMC 

B.  MTBF  and  MTBMA  by  two-digit  WUC 

C.  Re-test  OK  rates  by  two-digit  WUC 

D.  MMH/FH  by  two-digit  WUC 

E.  Break  rates 

F.  Fix  rates 

G.  Abort  rates — In-flight  and  before-flight 

2.  For  all  F-15s,  worldwide,  and  for  WUC  74  for  first  quarter  1993: 

-Overall  MTBMA 

-Listing  of  all  LRUs,  by  serial  number,  with  more  than  three  maintenance  actions  in  first  quarter  1993 
-Listing  of  all  LRUs,  by  serial  number,  with  MTBMA  less  than  overall  group  average 

3.  For  F-16,  part  number  1829003,  for  May  92  through  April  93,  list: 

Serial  number,  WUC  (14DPO  and  14DQO),  tail  number,  total  sorties,  total  maintenance  actions,  total 
maintenance  man-hours,  number  of  failures,  number  of  removals,  number  of  CNDs,  number  of  repairs, 
number  of  NRTSs,  narratives. 

4.  Algorithms  in  REMIS  and  REMISTALK  for  calculation  of  MC  rates,  MTBR,  MTBCF. 

Example  run  comparing  variables  drawn  from  REMIS  and  REMISTALK  to  see  whether  or  not  they  are 
the  same. 

Figure  D-1.  21  May  1993  Data  Request 


CAMS/REMIS  Data  Request — 26  May  1993 

1.  List  of  current  TCTOs  ftr  B-1  from  GCSAS 

2.  Seymour  Johnson  LANTIRN  and  backsbop  automatic  test  equipment  status 

3.  For  December  1992,  F-16C/D  at  ACC: 

Mission-capable  rates 

Flying  hours 
Sorties 

Maintenance  man-hours  per  flying  hour 

Mean  time  between  maintenance  actions — ^Typc  1,  Type  2,  Type  6 

Total  number  of  aircraft 

Total  man-hours 

Total  failures 

4.  Approved  and  actual  configuratton  for  any  tail  number  of  the  B-1 

5.  Frmn  CAMS,  number  of  transactions,  by  mcmdi,  January  1991  to  latest  available,  for  4th  Wing  at 
Seymour  Johnson  AFB  and  388th  Fighter  Wing  at  Hill  AFB 

Figuro  D>2.  26  May  1993  Data  Raqueat 
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Table  0>1.  Summary  of  Responses  to  First  Data  Request 

“■“RSiis  — —  — 

_ Item  Requested _ REMIS  Subsystem  TICARRS _ Notes _ 

1.  Mootbly,  Ocl  92-Apr  93,  for  F-15E,  4Ui  Wing:  See  Table  D-3. 

A.  Mis^-capeMeimes,  number  and  percent  of  Pinovided  excqit  EIMSURS  ftovided  March  and  April  93  status  data  not  available  in 
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TabI*  D>3.  Detail  on  Item  1  of  First  Data  Request 
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No  capability 


REMIS  did  not  provide  the  additional  Seymour  Johnson  data  we  asked  for  on 
MTBFand  MTBMA. 

SERIAL  NUMBER  TRACKING  AND  TRACKING  FOR  WARRANTIES 

Items  2  and  3  in  the  first  data  request  addressed  the  issue  of  serial  number  tracking. 
nCARRS  users  say  that  serial  number  tracking  is  necessary  for  identifying  bad  actors  and 
removing  them  from  the  system. 

As  background  information,  we  asked  for  data  on  MTBMA  for  F-15s,  and  both 
REMIS  and  TICARRS  provided  reasonable  data.  The  TICARRS  data  were  only  for  U.S. 
aircraft,  not  F-15s  worldwide,  as  requested. 

We  asked  for  all  serial  numbers  within  WUC  74  (fire  control)  that  had  more  than 
three  maintenance  actions  during  the  first  quarter  of  1993.  TICARRS  was  able  to  provide 
this;  REMIS  provided  an  alternative  report  that  listed  the  top  ten  man-hour  consuming 
systems  for  the  F-15C  by  WUC,  not  by  serial  number. 

We  also  asked  for  data  and  maintenance  narratives  on  the  F- 16  rotary  flap  actuator, 
a  part  that  is  tracked  by  TICARRS  for  warranty  purposes.  The  only  data  provided  by 
REMIS  were  maintenance  actions  and  man-hours  at  the  WUC  level.  TICARRS  provided 
68  complete  narratives  to  illustrate  its  capability.  The  REMIS  PMO  told  us  that  narratives 
can  be  retrieved  through  REMISTALK,  but  they  did  not  provide  us  with  a  sample.  The 
REMIS  developers  are  working  to  provide  retrieval  of  narratives  through  a  standard  PPS 
query  in  a  future  update,  currently  promised  for  first  quarter  of  FY  1994. 

ALGORITHMS  FOR  CALCULATION  OF  MAINTENANCE  PARAMETERS 

TICARRS  users  have  complained  that  the  algorithms  in  REMIS’s  PPS, 
EIMSURS,  and  REMISTALK  for  calculation  of  MC  rates,  MTBR,  and  MTBCF  are  not 
consistent  with  one  another.  The  REMIS  PMO’s  explanation  was  that  different  groups  of 
users  have  different  methods  of  calculating  these  variables.  Developers  have  decided  to 
wait  for  an  Air  Force  policy  on  what  the  algorithms  should  be,  and  they  will  then  be 
standardized  across  all  modules.  However,  discussions  within  the  Air  Force  about 
standardizing  algorithms  have  been  under  way  for  at  least  10  years.  There  is  no  reason  to 
think  that  a  standardized  Air  Force  policy  will  be  issued  in  the  near  future. 


GCSAS  ISSUES 


We  asked  for  several  items  from  GCSAS  in  the  second  request  (items  1,  2,  and  4). 
Although  it  is  not  yet  fully  operational,  we  wanted  to  get  a  sense  of  its  progress.  GCSAS 
was  able  to  provide  a  list  of  current  Time  Compliance  Technical  Orders  (TCTOs)  for  the 
B-1  (an  aircraft  not  supported  by  TICARRS),  but  was  not  able  to  provide  a  list  of  approved 
and  actual  configuration  for  the  same  aircraft,  because  the  actual  configuration  had  not  yet 
been  loaded.  Sample  screens  provided  to  us  looked  reasonable,  but  we  could  not  assess 
how  GCSAS  would  work  with  real  data. 

BASIC  F.16C/D  DATA 

In  our  second  data  request  (item  3),  we  asked  for  basic  F-16C/D  data.  Both 
systems  were  able  to  provide  most  of  the  data  we  wanted.  REMIS  supplied  only  on- 
equipment  man-hours,  not  total.  TICARRS  supplied  two  estimates  of  sorties,  one  that  was 
similar  to  the  REMIS  number  and  was  based  on  debrief  records  and  a  second,  which  users 
say  is  more  accurate,  that  was  higher  and  was  based  on  utilization  reports.  There  were 
some  major  differences  in  the  numbers. 

The  reported  maintenance  man-hours  per  flying  hour  from  REMISTALK  (4.71), 
was  inconsistent  with  the  reported  on-equipment  maintenance  man-hours  and  flying  hours 
(26104.2/10636.5  =  2.45).  The  TICARRS-reported  MMII/TH  of  10.37  was  consistent 
with  its  reported  components. 

In  other  cases  of  differences  in  magnitudes,  it  was  impossible  to  determine  which 
system — REMIS  or  TICARRS — ^had  the  correct  data. 

CAMS  DATA  REQUEST 

Item  5  in  our  data  request  was  directed  toward  CAMS  data  for  the  purpose  of  the 
Seymour  Johnson  assessment  and  is  not  relevant  to  our  assessment  of  REMIS. 

OBSERVATIONS 

In  several  cases,  the  REMIS  data  provided  were  for  a  different  time  period  than 
requested,  which  prevented  us  from  comparing  the  REMIS  data  with  the  TICARRS  data. 
In  one  instance,  TICARRS  provided  data  for  U.S.  F-15s  only,  rather  than  worldwide,  so 
we  could  not  compare  the  data  directly.  REMIS  provided  some  C-141  data  not  relevant  to 
our  request  (REMIS  output  on  the  C-141  is  obtained  through  an  interface  with  G081;  the 
C-141  fteet  does  not  use  CAMS.) 


With  regard  to  ease  of  use,  the  difficulty  of  pulling  data  from  the  three  parts  of 
REMIS  was  obvious.  Developers  of  the  different  parts  of  REMIS  had  to  confer  to 
determine  which  module  was  the  best  one  to  use. 

Furthermore,  the  people  who  retrieved  the  data  for  our  test  were  experts.  An 
average  user  would  probably  encounter  even  more  difficulty.  Aggregation — the  ability  to 
get  a  single  bottom-line  total  without  all  the  components — is  difficult  in  the  canned  queries 
in  the  EIMSURS  and  PPS.  In  many  cases,  REMIST ALK,  the  customized  inquiry  utility, 
was  the  only  way  to  retrieve  aggregate  data  without  a  lot  of  extraneous  output,  and 
REMISTALK  response  times  are  considerably  longer  than  those  for  standard  queries. 

The  TICARRS  data  were  retrieved  by  DRC  representatives  more  easily  and 
promptly  than  the  REMIS  data  could  be  retrieved  by  REMIS  developers.  The  TIC.\RRS 
data  were  provided  on  letter-sized  laser-printed  paper  and  included  a  roadmap  through  the 
data.  The  REMIS  response  was  on  wide-carriage  paper  and  was  very  difficult  to  follow. 

In  summary,  REMIS  demonstrated  some  basic  functions  reasonably  well  (e.g., 
mission-capable  rates)  but  did  not  consistently  demonstrate  other  basic  functions  such  as 
retrieval  of  total  maintenance  man-hours  and  mean  time  between  maintenance  actions.  This 
raises  questions  about  the  realism  of  the  schedule  for  full  implementation  of  REMIS. 
REMIS  did  not  demonstrate  serial-number  tracking  or  narrative  retrieval,  capabilities  that 
TICARRS  users  say  they  need.  Algorithms  for  calculation  of  key  maintenance  parameters 
are  not  uniform  within  REMIS.  IDA  personnel  found  the  TICARRS  output  much  easier  to 
follow  than  the  REMIS  output 
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THE  FUTURE  AIR  FORCE 

This  appendix  explores  the  adaptability  of  CAMS/REMIS  and  TICARRS  to  the 
future  operating  environment  and  condition  of  the  Air  Force.  Evaluating  adaptability  of 
these  systems  requires  a  vision  of  the  future  Air  Force. 

The  nature  of  the  future  Air  Force  with  respect  to  factors  that  affect  the  operation 
and  effectiveness  of  maintenance  information  systems  is  certainly  subject  to  debate. 
However,  that  vision  is  constrained  by  resources  and  technology.  Since  change  in  the  Air 
Force  does  not  occur  overnight,  our  vision  of  the  future  is  heavily  weighted  by  current 
Department  of  Defense  and  Air  Force  initiatives.  Alternative  visions  of  where  the  Air  Force 
will  be  in  the  future  are  not  necessarily  inconsistent  with  the  analysis  that  follows. 

Three  categories  of  factors  provide  the  proper  perspective  for  a  discussion  of  where 
the  Air  Force  will  be  in  the  next  seven  to  ten  years: 

•  operational  policies  and  practices, 

•  resource  constraints,  and 

•  technology  path  (the  ways  in  which  technology  is  expected  to  evolve). 

OPERATIONAL  POLICIES  AND  PRACTICES 

We  expect  that  operational  policies  and  practices  will  move  toward  less  reliance  on 
forward-deployed  forces,  centralized  control  of  air  assets  as  demonstrated  in  Desert 
Shield/Desert  Storm,  and  increased  reliance  on  fewer  levels  of  maintenance  to  support 
operating  units. 

The  bulk  of  U.S.  forces  will  be  CONUS-based,  and  will  be  required  to  deploy 
quickly  and  on  short  notice  to  locations  that  may  lack  significant  infrastructure  and  support 
This  deployment  environment  will  be  dominated  by  squadron-level  deployments  to  the 
theater  of  operation  and  requires  that  logistics  support  be  available  at  the  time  of 
deployment  be  easily  transportable,  ruggedized,  and  self-supporting. 

After  the  units  are  deployed,  the  air  campaign  will  be  undertaken  with  centralized 
command  and  control  that  directs  the  application  of  air  power  from  composite  wings  and 
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new  combinations  of  air  power  assets  (CONUS-based  bombers,  land-based  deployed 
aircraft  and  naval  aircraft).  As  in  Desert  Shield/Desert  Storm,  the  theater  air  commander 
will  develop  the  air  campaign  and  give  daily  operational  orders  to  the  units  under  his 
command. 

The  implications  for  logistics  support  are  that  the  theater  air  commander  will  need 
access  to  accurate  and  timely  information  on  the  readiness  of  air  assets  under  his  control, 
requiring  a  single  information  source  (central  data  base)  to  speed  complex  information 
flows.  This  deployment  environment  also  requires  that  information  systems  easily  interface 
with  each  other,  offering  a  sense  of  seamlessness  in  supply,  distribution,  maintenance,  and 
transportation.  CONUS-based  support  for  this  deployment  scenario  will  depend  on  rapid 
and  effective  communications  with  the  theater  of  operation. 

Finally,  the  operational  organization  of  the  Air  Force  may  change  in  ways  that  make 
some  kinds  of  information  more  important  than  they  used  to  be.  The  adoption  of  composite 
wings  with  several  different  types  of  aircraft  and  increased  reliance  on  two-level 
maintenance  to  support  the  different  aircraft  types  is  an  example  of  such  a  change.  In 
Coronet  Deuce,  the  application  of  two-level  maintenance  for  avionics  and  engines  to 
selected  F-16  units,  the  Air  Force  identified  a  need  for  asset  visibility  to  provide  support  to 
the  operational  units.  Asset  visibility  includes  having  information  to  track  demands  for 
spare  parts,  maintenance  actions,  transportation  and  processing  of  repairable  assets  at  the 
depots,  identification  of  bad-actor  components,  and  testing  of  software  incompatibilities 
between  levels  of  maintenance.^ 

The  effect  of  these  policies  is  that  the  maintenance  data  system  will  need  to  support 
the  deployment  of  squadron-level  units  and  to  provide  information  to  the  theater  air 
commander  to  support  execution  of  the  air  campaign,  to  support  deployed  units  in 
peacetime  and  conflict,  and  to  support  the  application  of  two-level  maintenance  to  a  wider 
segment  of  weapon  systems. 

RESOURCE  CONSTRAINTS 

Resource  constraints  mean  that  the  future  Air  Force  will  have  a  significantly  smaller 
force  structure  and  that  operating  with  reduced  support  and  personnel  in  the  logistics  tail 
will  be  desirable.  Two-level  maintenance,  a  way  of  cutting  the  tail,  will  place  a  premium  on 


^  Both  two-level  maintenance  and  cmnposite  wings  involve  performing  more  maintenance  at  looaions 
other  than  the  base  where  a  part  was  found  to  fail.  Quick  and  effective  access  to  reliable  fleet-wide 
maintenance  history  data  may  be  more  important  under  such  drcumstances. 
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dat?>  in  the  maintenance  data  system.  Maintenance  histories  and  other  information  will  have 
to  be  more  accessible  and  accurate  than  in  the  past  because  the  Air  Force  will  not  have  the 
opportunity  to  buy  excess  spare  parts  as  a  deliberate  response  to  uncertainty  about  ..ctual 
requirements.  Better  information  on  the  status  of  assets,  including  their  location  while  they 
are  moving  through  the  transportation  system,  will  be  needed.  Fewer  assets  in  the  system 
also  requires  that  those  assets  be  highly  reliable  and  easily  maintainable  and  supportable. 
This  will  place  increased  emphasis  on  being  able  to  identify  support  problems  and  justify 
the  resource  expenditures  to  correct  the  problems.  The  maintenance  data  system  will  have 
to  provide  the  information  to  identify  problems  and  justify  the  resources  to  fix  them. 

TECHNOLOGY  PATH 

The  third  consideration  is  the  technology  path,  that  is,  what  kinds  of  technological 
change  can  be  expected,  notably  in  the  weapon  systems  being  supported  and  in  the 
information  system  development  and  operational  environment.  The  following  subsections 
address  some  of  characteristics  of  technological  change  that  seem  realistic. 

Emerging  Weapon  System  Technology 

Weapon  systems  will  become  increasingly  complex,  software-intensive,  and 
reliable.  Increased  emphasis  will  be  placed  on  providing  built-in  test  capability  in  the 
aircraft  to  diagnose  the  nature  of  equipment  problems.  The  maintenance  data  system  should 
be  able  to  accept  electronic  information  on  failure  modes  and  circumstances  directly  from 
the  aircraft  Similarly,  test  equipment  at  all  levels  of  maintenance  should  be  able  to 
communicate  (send  and  receive  information)  directly  to  the  maintenance  data  system. 

Weapon  systems  such  as  the  F-22  will  rely  on  integrated  diagnostics  to  ensure 
support  of  the  weapon  system.  Integrated  diagnostics  will  be  needed  to  address  support 
problems  with  the  evolving  integrated  digital  avionics  and  distributed  sensor  systems 
[electro-optical  and  radio  frequency  (RF)  technologies].  In  the  case  of  digital  avionics,  the 
built-in  test/self-test  (BIT/ST)  capability  will  generate  large  amounts  of  detailed  information 
that  will  be  provided  to  a  maintenance  data  system.  The  information  will  be  needed  to 
assess  the  health  of  the  integrated  digital  avionics  system  and  to  determine  when  to  replace 
line  replaceable  modules  that  are  characterized  by  graceful  degradation.  Evolving  RF 
systems  include  solid  state,  active  aperture,  and  phased-array  technologies  that  use  large 
numbers  of  transmit/receive  units  and  have  significantly  improved  reliability.  The  RF 
systems  will  also  be  characterized  by  graceful  degradation  and  will  rely  on  BFT/ST  to  help 
detomine  wten  sufikient  transmit/receive  units  have  failed  and  affect  system  performance. 


This  process  will  generate  information  that  will  be  stored  in  a  maintenance  data  system. 
This  information  will  assist  in  determining  when  systems  are  failing  to  meet  performance 
requirements,  assist  in  trouble-shooting  problem  systems,  and  allow  proper  maintenance 
decisions  relative  to  bad-actor  identification  efforts. 

The  amount  and  complexity  of  software  and  integrated  circuits,  driven  by  advances 
in  micro-circuitry,  will  increase  dramatically,  placing  new  demands  on  configuration 
management,  testing,  and  repair.  Maintenance  technologies  will  require  more  emphasis  on 
software  support  as  weapon  systems  become  more  software-intensive.  The  software¬ 
intensive  systems  of  the  future  will  raise  issues  of  how  to  measure  software  effectiveness, 
not  only  operationally  but  environmentally.  How  does  the  reliability  of  the  software  relate 
to  the  reliability  of  the  hardware  platform?  The  maintenance  data  systems  will  collect 
information  on  the  performance  of  the  systems  to  help  segregate  performance  problems 
among  the  hardware,  operational  flight  software,  and  maintenance  software.  The 
maintenance  data  system  will  provide  the  link  from  the  problems  that  are  experienced 
during  operation  of  the  weapon  system  to  the  test  equipment  performing  maintenance  tests 
and  to  the  maintainer  operating  the  testing  system.  Also,  the  operating  units.  Major 
Commands,  and  engineers  or  technicians  monitoring  the  health  of  a  system  will  rely  on  this 
information  to  identify  bad-actor  components  and  candidate  systems  for  modifications  and 
software  improvements. 

Expert  systems  that  aid  in  the  identification,  diagnosis,  and  repair  of  weapon 
systems  will  be  available.  Automated  maintenance  aids,  such  as  Integrated  Maintenance 
Information  System  (IMIS),  that  use  expert-system  techniques  may  become  widely 
available,  replacing  technical  manuals,  and  aiding  in  troubleshooting  the  weapon  system.  In 
some  cases,  expert-system  software  may  aid  the  evolution  of  the  testing  strategy  and 
improve  test  programs  through  the  use  of  artificial  intelligence  techniques.  These 
techniques  will  partially  rely  on  data  in  the  maintenance  information  system  such  as  failure 
histories,  maintenance  histories,  and  narrative  accounts  of  the  operational  or  testing 
problems.  In  either  case,  these  aids  will  benefit  from  direct  access  to  maintenance-history 
data  that  should  reside  in  the  maintenance  information  system. 

Evolving  Information  System  Technologies 

Over  the  past  several  years,  we  have  witnessed  significant  improvements  in 
price/performance  ratios  in  technological  areas  that  are  key  to  the  future  design  of 
information  systems.  For  example: 
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•  Solid-state  technology  continues  to  improve  and  large  volumes  of  even  higher 
densities  and  faster  devices  are  being  produced  with  each  new  generation  of 
chip.  Reduced  power  requirements,  new  packaging,  and  better  cooling 
techniques  contribute  to  the  ability  to  use  the  chips  in  a  wide  variety  of 
applications  and  environments.  High-technology  production  processes  have 
resulted  in  high-yield,  low-cost  chip  production. 

•  Communication  technologies  are  taking  advantage  of  fiber  optics  to  provide 
higher  bandwidth  facilities  at  lower  cost,  making  it  possible  to  reliably  transmit 
and  receive  increasingly  larger  quantities  of  high-resoludon  digital  data  at  low 
cost. 

•  Computer  technology  is  taking  advantage  of  advances  in  communications  and 
solid-state  technology  in  several  ways.  Faster  and  more  dense  solid-state 
devices  lead  to  faster  and  more  powerful  general-purpose  computers. 
Computers  that  used  to  be  considered  large  scale  are  now  small  in  size  and 
power  consumption,  and  even  larger  in  computing  power.  According  to 
technology  experts,  this  trend  is  likely  to  continue  for  the  remainder  of  this 
decade,  resulting  in  significantly  lower  price/performance  ratios,  and 
computers  that  will  provide  the  equivalent  of  large-scale  power  at  mini¬ 
computer  prices.  Personal  computers,  only  recently  becoming  useful  and 
popular  as  laptops,  are  now  headed  for  hand-held  model  production.  Hand¬ 
held  computers  coupled  with  a  short-range  radio  or  cellular  transmitter  and  the 
use  of  multimedia  technology  (see  below)  offer  new  opportunity  for 
dramatically  more  efficient  and  reliable  data  collection  and  workload  scheduling 
and  monitoring. 

•  Data  storage  technology  has  developed  along  two  fronts.  Magnetic  storage  is 
continuing  on  the  path  of  increased  density  and  smaller  footprint  devices  at  all 
levels,  resulting  in  the  ability  to  economically  store  large  volumes  of  readily 
accessible  data.  Optic  technology  offers  “write  once,  read  many”  (WORM) 
technology,  which  is  being  used  for  storing  huge  quantities  of  text  and 
graphical  data  on  compact  disk,  leading  to  more  effective  distribution  of 
reference  materials,  documents,  and  the  like. 

•  Software  engineering  and  programming  technology,  while  seemingly  not 
moving  as  fast  as  other  technological  areas,  has  in  fact  been  able  to  take 
advantage  of  the  newest  hardware  technologies  and  has  made  significant 
progress  towards  instituting  predictable  and  productive  engineering-like 
disciplines  and  technologies.  Instituting  programming  process  management 
and  measurement  disciplines  has  resulted  in  significantly  improved 
productivity,  quality,  and  adherence  to  schedules  of  software  development 
iK^tivities.  The  development  of  object-oriented  software  and  databases  offers 
opportunities  for  software  reuse  and  improved  quality.  Expert  systems 
technology,  or  rules-based  softv.'are  systems,  are  becoming  more  prevalent  in 
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applications  involving  diagnostic  or  decision  assistance.  The  availability  of 
affordable  communications  and  storage,  and  Iov^?-cost  computer  chips  has  lead 
to  both  the  development  of  economically  feasible  and  highly  user-friendly 
graphical  interfaces  and  the  introduction  of  mixed  media  (sound,  voice, 
imaging,  graphics,  and  text)  to  communicate  with  the  users.  The  software 
industry,  realizing  the  road  to  success  is  to  allow  for  interchangeability,  is 
racing  towards  supporting  open  systems  and  industry  standards. 

While  the  technology  will  continue  to  make  advances,  prudent  managers  of  existing 
information  systems  are  establishing  strategies  that  will  allow  them  to  take  advantage  of  the 
technology  as  it  matures  without  revamping  the  entire  information  system.  They  are 
looking  to  evolve  to  the  new  technologies  rather  than  revolutionize  the  process  and  systems 
currently  in  place. 

The  weapons  maintenance  information  system  of  the  future  needs  to  have  the 
following  strategically  important  features; 

•  deploy  at  the  squadron  level; 

•  adapt  to  new  weapon  systems,  maintenance  technology,  tools  and  procedures; 

•  interface  with  other  systems; 

•  adapt  to  Air  Force  organizational  changes  and  processes; 

•  provide  accurate  and  timely  data; 

•  provide  user-friendly  (productive  and  efficient)  data  collection  and  data  access; 

•  be  economically  feasible  (affordable)  to  operate; 

•  accommodate  evolving  information  system  and  data  automation  technologies; 

•  provide  centralized  data  for  fleet-wide  perspective  of  status,  reliability  and 
maintainabili^  data;  and 

•  sustain  high  system  reliability  and  performance  characteristics. 

Figures  E-1  and  E-2  illustrate  a  weapons  maintenance  configuration  that  takes 
advantage  of  current  information  systems  technology  and  has  those  features. 

The  configuration  shown  in  Figure  E-1  is  based  on  providing  squadron-level 
modularity  by  using  local  area  networks  (LANS)  and  client-server  data  base  (CSDB) 
systems.  Each  squadron  has  its  own  CSDB,  LAN,  work  stations,  and  test  stations  to 
record  and  evaluate  maintenance  data.  The  squadron-level  database  has  access  to  and  may 
be  accessed  by  both  a  wing  CSDB  and  a  central  data  repository.  Essential  fleet-wide  data 
are  gathered  from  the  squadron  by  the  repository  system.  The  repository  system  is 
immediately  available  to  the  squadron  user  or  for  fleet-wide  data.  Communicaticm  betweoi 


the  squadron  data  base  (SQDB)  and  the  repository  is  via  high-speed  fiber-optic 
communication  links  on  wide  area  or  local  area  networks  (WAN/LAN).  This  allows  the 
repository  to  not  only  transmit  and  receive  data  from  the  nodes  on  the  link,  but  also 
maintain  the  programs  and  operation  of  the  squadron  CSDB,  obviating  the  need  for  a 
computer  operations  staff  at  each  squadron.  The  squadron  has  access  to  similar  client 
servers  or  gateways  on  the  link,  so  that  appropriate  data  may  be  exchanged  between 
complementary  systems  [e.g..  Comprehensive  Engine  Management  System  (CEMS),  fuel, 
and  Combat  Communications  System  (CAS-B)].  In  turn,  the  central  repository  will 
support  major  functional  units  such  as  repair  depots  and  technical  data  distribution  centers 
through  direct  links  or  WAN  communication  systems.  The  technical  data  distribution 
centers  prepare,  maintain,  and  distribute  optical  compact  disks  to  the  squadrons  based  on 
weapon  and  equipment  ownership. 


Figur*  E-1.  Futur*  Wtapont  Maintonanca  Information  Syatam 


Dq)loyed  squadrons  also  use  the  CSDB  LAN  system  configuration,  but  are  able  to 
sustain  operation  without  continuous  communication  to  tire  central  repository,  since  the 
CSDB  contains  all  the  data  relevant  to  a  deployed  situatitm.  The  deployed  squadron  will 


periodically  use  satellite  links  to  communicate  essential  data  to  and  from  the  central 
repository. 


Figura  E>2.  Futura  Squadron*L«v«l  Woapons  Maintenanca  Information  Systam 

At  the  squadron  level  (Hgure  E-2},  LANs  are  used  to  communicate  with  a  variety 
of  work  stations  and  test  stations,  all  of  which  use  a  standard  LAN  adapter  and  common 
serial/parallel  interface.  The  addition  of  new  test  or  terminal  equipment  is  facilitated  by 
using  standard  interfaces  and  common  communication  protocols.  Personnel  entering 
maintenance  data  are  provided  data-entty  windows  that  allow  selection  of  appropriate  data 
(e.g.,  part-serial  numbers)  from  lists  rather  than  by  entering  the  information  manually.  This 
practice  will  reduce  both  data-entry  errors  and  the  time  technicians  spend  at  the  terminal. 
Forthcoming  diagnostic  data-coUection  devices,  both  hand-held  and  integrated,  interface 
with  the  LAN  adapter  to  send  data  to  the  CSDB  and  to  be  evaluated  by  diagnostic  expert 
^sterns  at  the  work  stations.  Powerful  personal  computer  terminals  have  graphic  displays 
and  voice  response  units  that  allow  not  only  the  data  analyzers  to  easily  see  the  trends  of 
maintenance  dma,  but  provide  help  and  guidance  to  the  novice  and  those  in  training. 
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EFFECTS  OF  INFORMATION  SYSTEMS  ON 
WEAPON  SYSTEM  PERFORMANCE 

A.  OBJECTIVE 

This  appendix  describes  the  effect  (or  lack  of  effect)  of  CAMS/REMIS  and 
nCARRS  on  weapon  system  performance  and  maintenance-related  support.  The  objective 
is  to  determine  if  these  information  systems  significantly  affect  the  operational  and  support 
environment  If  they  do,  the  comparison  of  CAMS/REMIS  to  TICARRS  takes  on  mission- 
critical  dimensions;  if  not  these  information  systems’  evaluation  moves  one  step  closer  to  a 
pure  cost  comparison. 

A  comparison  is  made  of  performance  and  support  over  the  period  of  time  when 
TICARRS  users  were  required  to  switch  from  direct  entry  into  TICARRS  to  entry  through 
CAMS  and  then  into  TICARRS.  Weapon  systems  supported  by  TICARRS  are  the  focus 
here  but  two  non-TICARRS  weapon  systems  are  included  to  control  for  any  simultaneous 
Air  Force  activities. 

B.  APPROACH 

The  approach  is  to  use  a  time  series  of  weapon  system  and  product  performance 
data  for  tactical  weapon  systems  to  create  a  baseline  or  historical  snapshot  from  which  the 
conversion  from  direct-entry  TICARRS  reporting  to  CAMS-entry  TICARRS  reporting  can 
be  evaluated.  The  CAMS  conversion  dates  are  then  “overlaid”  onto  these  trends  to 
determine  if  there  are  significant  differences  in  pre-  and  post-CAMS  conversion 
performance.  Formal  statistical  tests  are  used  to  evaluate  these  time  trends. 

Data  requests  were  made  of  Dynamics  Research  Corporation  (DRC)  and  the 
CAMS/REMIS  Program  Mana^ment  Office  (PMO)  to  provide  the  following  infcamatitxi: 

*  inventory,  status,  and  utilization  data  for  tactical  weapon  systems  over  the 
period  1982  through  1992  and 
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•  product  performance  data,  including  failures,  demands,  maintenance  man¬ 
hours,  and  so  on,  by  two-digit  work  unit  code  (WUC)  for  these  weapon 
systems. 

DRC  provided  these  data  for  all  F- 15  and  F-16  weapon  systems  on  the  TICARRS 
system.  TTie  CAMS/REMIS  PMO  provided  recent  weapon  system  performance  data  only, 
for  F-1 1  Is,  F-4Es,  and  other  weapon  systems.  A  similar  request  to  Air  Combat  Command 
(ACC)  provided  hard  copies  of  more  historical  data  for  non-TICARRS  weapon  systems. 

The  weapon  system  performance  data  were  used  to  calculate  a  measure  of 
operational  readiness,  the  mission-capable  rate  (MC),  defined  as 

MC  =  (FMC  +  PMC)/possessed  hours, 

where  FMC  means  fully  mission  capable  and  PMC  means  partially  mission  capable. 

The  product  performance  data  supplied  for  the  F-16  and  F-1 5  systems  by  DRC 
were  used  to  calculate  mean  time  between  failures  (MTBF)  for  types  1,  2,  and  6  failures 
and  maintenance  man-hours  per  flying  hour  (MMH/FH). 

Two  models  are  developed  and  tested  to  evaluate  weapon  system  performance  in 
the  context  of  the  information  systems.  The  Erst  model  focuses  solely  on  the  F-16  weapon 
system  and  its  conversion  from  TICARRS  direct  entry  to  entry  through  CAMS  that 
occurred  during  the  period  late  1988  through  1989.  A  regression  model  of  the  following 
form  is  developed: 

y  (MC)  =  f(CAMS.  T86-T92,  e), 

where 

MC  =  the  mission  capable  rate; 

CAMS  =  the  (wing-specific)  conversion  date  from  TICARRS  to  CAMS; 

T86-T92  =  a  series  of  dummy  vaiiables  taking  the  value  1  in  the  year  indicated  and 
0  otherwise;  and 

e  =  the  error  term. 

The  second  model  focuses  on  three  weapon  systems,  the  F-16,  F-1 11,  and  F-4E, 
aU  of  which  had  increasing  (or  at  least  non-decreasing)  MC  rates  over  the  relevant  period  of 
this  evaluation.  The  key  independent  variable  in  all  these  equations  is  a  period  measure  that 
identifies  three  time  periods:  (1)  the  time  before  conversion  to  CAMS,  (2)  tite  period  during 
which  all  F-16  bases  were  converted  to  CAMS  entry,  and  (3)  the  period  after  the 
conversion. 


A  series  of  regression  models  were  developed  with  the  following  functional  form: 


y(MC) 

=  f(Pi,  e). 

(F-1) 

y  (MTBFj) 

=  f(Pi.  e). 

(F-2) 

y  (MMH/FH) 

=  f(Pi.e) 

(F-3) 

where, 

MC 

MTBFi 


MMH/FH 

Pi 


e 


the  mission  capable  rate,  defined  above; 

the  inherent  failure  rate  for  selected  two-digit  WUCs  (engines  and 
avionics); 

the  maintenance  man-hour  measure  defined  above; 

a  set  of  three  0-1  dummy  variables  where  Pi  =  1  from  January  1985 
through  September  1988,  0  otherwise;  P2  =  1  from  October  1988 
through  May  1990, 0  otherwise;  and  P3  =  1  from  June  1990  through 
the  end  of  the  time  series,  0  otherwise;  and 

the  error  term. 


Equation  (F-1)  was  run  separately  for  the  F-16,  F-4E,  and  F-1 1 1  weapon  systems. 
The  coefficients  on  the  dummy  variables  were  compared  across  the  different  weapon 
systems.  The  latter  weapon  systems  were  used  in  the  analysis  to  provide  a  control  group 
against  which  the  TICARRS  reporting  weapon  system  could  be  evaluated.  If  the  F-16 
results  are  found  to  be  significantly  different  from  the  other  two  aircraft  types,  the 
conversion  to  CAMS  may  have  had  an  impact  on  weapon  system  performance.  If  not,  cost 
becomes  a  more  important  driver  of  the  CAMS/REMIS  and  TICARRS  comparison. 
Equations  (F-2)  and  (F-3)  were  run  for  F-16  data  only.  These  equations  measure  tlm 
impact  of  the  CAMS  conversion  on  product  performance  support  No  similar  data  for  the 
other  two  weapon  systems  were  available  for  comparable  testing.  Ordinary  least  squares 
(OLS)  regression  techniques  were  used.  In  all  three  cases  [equations  (F-1),  (F-2),  and 
(F-3)],  the  pre-CAMS  period  (dummy  variable  Pi)  was  chosen  as  the  omitted  category. 


C.  DATA  ANALYSIS 

Data  were  evaluated  for  all  active  Air  Force  F-16As  and  F-16Cs  on  the  TICARRS 
system  in  the  first  model  and  for  F-16s,  F-1 11s,  and  F-4Es  on  the  CAMS/REMIS  system 
for  the  second  model,  over  the  period  1985  to  1992.  The  analysis  is  conducted  by  wing 
(and  by  mission,  design,  series  in  the  second  model).  The  tables  provided  are  examples  of 
the  basic  ^tem  data  used  in  this  analysis. 


After  some  casual  observation  of  the  data,  regression  models  were  implemented  and 
evaluated.  OLS  estimates  were  derived  and  the  equations  were  evaluated  for  statistical 
estimation  problems.  The  results  for  the  first  model  are  shown  in  Table  F-1. 


Table  F-1.  Model  1:  F-16 


Wc'^pon  System: 

F-16 

Dependent  Variable: 

MC 

Indeuendent  Variables: 

Coefficient  (T-Ratio) 

Intercept 

0.83  (84.05) 

CAMS 

0.006  (.462) 

T86 

-0.016  (-1231) 

T87 

0.013  (0.999) 

T88 

0.063  (4.84) 

T89 

0.056  (3.45) 

T90 

0.067  (3.98) 

T91 

0.068  (3.86) 

T92 

0.064  (3.57) 

Adjusted  R2  ss 

0.0813 

The  coefficients  on  T88  to  T92  are  statistically  significant  at  the  ,99  level  The 
coefficient  on  CAMS  is  not  statistically  significant. 

The  results  of  this  first  model  indicate  that  the  conversion  to  CAMS  did  not 
significantly  affect  the  MC  rate  of  the  F-16.  While  the  MC  rates  were  changing  over  the 
seven  years,  those  changes  are  indicative  of  time  changes,  not  any  conversion  from 
TICARRS  entry  to  CAMS  entry.  The  results  show  an  MC  rate  of  about  83  percent  with 
statistically  significant  increases  over  the  years  1988  to  1992.  The  CAMS  variable  is  not  at 
all  significant  to  changes  in  the  MC  rate. 

The  results  for  the  second  model  are  shown  in  Table  F-2.  All  of  the  coefficient 
estimates  are  significant  at  the  .99  level  except  P2  for  the  F-1 1 1,  which  is  significant  at  the 
.975  level. 


Table  F-2.  Model  2:  F-16,  F-111.  and  F-4E 


Weqxm  System: 

F-16 

F-111 

F-4E 

Dqxndent  VariaUe: 

MC 

MC 

MC 

Ckiefficient  (T-Ratio) 

Coefficient  (T-Ratio) 

Coeffident  (T-RaUo) 

laiaccpt 

0.84  (18350) 

0.68  (94.10) 

0.85  (22658) 

P2 

0.051  a.06) 

0.030  CIS6) 

0.021  (3.06) 

P3 

0.062  (9.67) 

0.093  (9.10) 

0.032  (4.14) 

Adjusted  R2« 

0.0656 

0260 

0.082 

The  results  of  this  series  of  regression  models  indicate  that  there  is  no  statistically 
significant  difference  in  the  MC  rates  of  the  three  weapon  systems.  The  estimates  for  P3 
show  the  F- 16  value  between  the  value  for  the  other  two  weapon  systems,  and  all  are 
significant  The  conclusion  is  that  conversion  to  CAMS  did  not  significantly  affect  weapon 
system  performance. 

A  number  of  regression  models  were  run  to  determine  if  the  conversion  to  CAMS 
affects  MTBF  or  MMH/FH  for  WUC  74,  fire  control  and  inertial  navigation  system.  For 
the  MTBF  measure,  a  CAMS  conversion  indicator  variable  and  variables  designating  each 
year  was  implemented.  The  CAMS  conversion  variable  was  not  statistically  significant  A 
second  model  that  employed  a  time  trend  was  not  statistically  significant  The  results  of  a 
third  model  using  dummy  variables  that  signify  time  periods  (P2  and  P3  as  previously 
described)  are  shown  in  Figure  F-3.  For  the  MMH/FH  measure,  the  equation  with  the 
CAMS  conversion  dummy  showed  no  statistical  significance  for  the  CAMS  variable. 
Relative  to  1987,  the  years  1988  through  1991  show  statistically  significant  (negative) 
coefficients,  although  the  values  of  the  coefficients  for  1989, 1990,  and  1991  varied  little, 
indicating  no  significant  difference  in  MMH/FH  between  those  three  years.  The  time 
periods  version  (P2  and  P3)  is  shown  below. 


Figure  F<3.  Model  3:  F-16  (MTBF)  and  F-16  (MMH/FH) 


Weapon  System; 

F-16 

F-16 

Dependent  Variable: 

MC 

MC 

Coefficient  (T-Ratio) 

Coefficient  (T-Ratio) 

Inteioqx 

2.325  (10.86) 

4.06(3.63) 

P2 

-0.824  (2.17) 

-2.850  (1.58) 

P3 

-0.135  (0.45) 

-0.382  (028) 

Adjusted  « 

0.001 

0.007 

Those  two  equations  (tetermine  whether  there  is  any  statistical  significance  between 
the  CAMS  and  post-CAMS  conversion  periods  relative  to  the  pre-CAMS  period  using  two 
measures  of  product  performance,  MTBF  and  MMH/FH. 

Both  intercepts  are  significant  and  P2  in  the  MTBF  equation  is  also.  There  is  a 
definite  decline  in  MTBF  in  the  1988-1989  praod.  However,  no  significant  change  in  MC 
rate  occurred.  The  tracking  of  fiying  hours  is  so  closely  monitored  in  the  Air  Force  diat  one 
can  assume  that  the  observed  trend  is  not  doe  to  a  decline  in  fiying  hours.  Observation  of  a 
series  of  plots  confirms  this  assumption.  A  logical  explanation  might  be  an  increase  in  the 
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number  of  transactions  entered  into  the  system  during  the  conversion  period.  However, 
previous  views  of  conversions  to  CAMS  saw  a  drop  in  transactions  reported  in  the  system. 
Such  a  drop  would  produce  a  positive  effect  on  MTBF,  not  negative.  It  appears  that  this 
finding  is  time-specific  and  not  related  to  the  CAMS  conversion. 

D.  RESULTS 

'Ihe  results  of  the  analysis  presented  above  indicate  that  consistently  across  bases 
and  wings,  the  conversion  to  CAMS-entry  from  direct-entry  TICARRS  for  the  F-16 
weapon  system  produced  no  significant  change  in  weapon  system  performance.  Compared 
to  the  F-1 1 1  and  F-4E,  there  are  no  differences  in  MC  rates  during  the  period  of  conversion 
to  CAMS  entry  for  the  F-16.  Regarding  product  performance  of  a  selected  WUC  for  the 
F-16  aircraft,  the  results  indicate  that  the  conversion  itself  did  not  affect  MTBF  or 
MMH/FH,  although  some  time-sensitive  trends  did  occur. 
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ACC 

Air  Combat  Command 

ADP 

automatic  data  processing 

AFB 

Air  Force  Base 

APR 

Air  Force  Regulation 

AFRES 

Air  Force  Reserve 

AFTO 

Air  Force  T^hnical  Order 

AGE 

aerospace  ground  equipment 

ALC 

Air  Logistics  Center 

AME 

Alternate  Mission  Equipment 

AMOC 

Automated  Maintenance  Operations  Center 

ANG 

Air  National  Guard 

ATC 

air  traffic  control 

ATE 

automatic  test  equipment 

ATERS 

Automatic  Test  Equipment  Reporting  System 

BAQ 

Basic  Allowance  for  Quarters 

BIT 

built-in  t^t 

BP 

budget  program 

CAMS 

Core  Automated  Maintenance  System 

CDB 

Central  Data  Base 

CDS 

Centralued  Data  Syst^ 

CEMS 

Comprehensive  Engine  Management  System 

CCffiO. 

Common  Business  Oriented  Language 

CONUS 

continental  United  States 

CSDS 

cltent-server  data  base  system 

CSRD 

Computer  System  Requirements  Document 

DBM 

data  base  manager 

DCP 

Data  Communication  Processor 

DDN 

Defense  Data  Network 

DIREP 

l^fEculty  Report  or  Discrepancy  Report 

Dc£> 

Department  of  Defense 

DRC 

Dynamics  Research  Corporation 

EIMSURS 

FDWW 

FMC 

FOC 

G&A 

GAO 

GCSAS 

ICE 

ID 

IDA 

IWSM 

JCN 

JDD 

Kbps 

LAN 

LANTIRN 

LCS 

LRU 

MAISRC 

MAJCOM 

MC 

MDC 

MIPS 

MMH/FH 

MMICS 

MOC 

MTBCF 

MTBF 

MIBMA 

MTBR 

NATO 

NCC 

NRTS 

O&M 

O&S 

OMB 


Equipment  Inventory,  Multiple  Status  and  Utilization  Reporting  Subsystem 

Functional  Disconnect  Work-Around  Write-Up 

fully  mission  capable 

full  operational  capability 

general  and  administration 

General  Accounting  Office 

Generic  Configuration  Status  Accounting  System 

Independent  Cost  Estimate 

identification 

Institute  for  Defense  Analyses 

Integrated  Weapon  System  Management 

job-control  number 

Job  Data  Documentation 

kilobytes  per  second 

local  area  network 

Low  Altitude  Navigation  and  Tracking  Infrared  System  for  Night 
Litton  Computer  Services 
line  replaceable  unit 

Major  Automated  Information  System  Review  Council 

Major  Command 

mission  capable 

Maintenance  Data  Collection 

million  instructions  per  second 

maintenance  man-hours  per  flying  hour 

Maintenance  Management  Information  Control  System 

Maintenance  Operations  Center 

mean  time  between  critical  failures 

mean  time  between  failures 

mean  time  between  maintenance  actions 

mean  time  between  repairs 

North  Atlantic  Treaty  Organization 

Network  Control  Center 

not  repairable  this  station 

operations  and  maintenance 

operating  and  support 

Office  of  Managemoit  and  Budget 
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PACAF 

PC 

PMC 

PMO 

POE 

PPS 

PQDR 

C3T&E 

QT&E 

R&M 

RAM 

RDA 

RECFUn 

REMIS 

RF 

RPC 

SBLC 

SBSS 

SDS 

SE 

SLOC 

SPO 

SPQR 

SSC 

SSI 

SSO 

ST 

Ta 

Tcro 

nCARRS 

USAFE 

WAN 

WORM 

WUC 


Pacific  Air  Force 
personal  computer 
partially  mission  capable 
Program  Management  Office 
Program  Office  Estimate 
Product  Performance  Subsystem 
Product  Quality  Deficiency  Report 
qualification  testing  and  evaluation 
qualification  testing  and  evaluation 
reliability  and  maintainability 
random  access  memory 
Reportable  Database  Area 
Recovered  Functionality 

Reliability  and  Maintainability  Information  System 

radio  frequency 

Re^onal  Processing  Center 

Standard  Base  Level  Computer 

Standard  Base  Supply  System 

Smart  Data  System 

support  equipment 

source  lines  of  code 

System  Program  Office 

System  Product  Quality  Reporting 

Standard  Systems  Center 

system-to-system  input 

system-to-system  output 

self-test 

time  change  item 

Hme  Compliance  Technical  Order 

Tactical  Interim  Cote  Automated  Maintenance  System  and  Reliability  and 
Maintainability  Information  System  Reporting  System 

U.S.  Air  Forces  Europe 

wide  area  network 

write  once,  read  many 

work  unit  code 
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